Python development: custom web scraping and data processing

Scrapers and macros: what people order most

People ask for Excel macros when the same report is assembled by hand every week: a macro turns half a day of work into one button. They ask for a scraper when the data sits on a website but there is no export — prices, catalogues, contacts, listings.

You generally cannot buy a scraper off the shelf: every site is built differently, and a working solution is always written for one specific source. In exchange, once delivered the script is yours together with its source code — there is nothing to pay for a second time.

Scraping is priced by the source, not by the number of pages: a simple catalogue starts from ₽3,000 (≈ $36); a site with a login and bot protection costs more. The price is always named before the work starts, as a separate line in the estimate. Reports come in whatever form you actually work with: Excel, Google Sheets, CSV or a ready database.

Custom data processing starts where a spreadsheet no longer fits in your head: exports from different systems, dates in different formats, the same customer duplicated across two databases. A script for that kind of job pays for itself within a couple of weeks — what takes a person all day, a machine counts in a minute and does not get tired by evening. Getting data out of a closed system is realistic too: where there is no ready export, we take it through an API or work through the interface. The price depends on how predictable the source is: one tidy table is one job; ten files from different departments with hand-made corrections is another entirely.

What automation costs and what the job includes

The question is almost always “how much does a script cost”, and the answer starts not with a number but with a question back: which routine exactly are we removing, and how much time does it eat now. A task that takes half a day a week and one that takes half a day every day mean different money and different deadlines, even when the code turns out similar.

How the work begins

Not with code. First we write it out in plain words: what the person does by hand, how many times a week, where the data comes from and where the result goes. After a description like that, some tasks turn out simpler than they looked — sometimes a spreadsheet formula or a setting inside the service is enough. We say so plainly: selling a script where a setting would do is bad for us first of all, because nobody recommends that kind of work to a friend.

What gets ordered most often

Pulling a report from several sources into one table, when the data lives in three places and is merged by hand every time. A catalogue scraper that refreshes prices on a schedule. A Telegram bot that takes an enquiry and files it into a spreadsheet without a manager touching it. Moving data between services through an API, so the same thing is not copied twice. Clearing duplicates out of a database — a week by hand, a minute by script.

What the price is made of

Not the number of lines of code, but how predictable the source is. A tidy export with consistent columns is one job. Ten files from different departments, with dates in three formats and half the names misspelled, is another: most of the time goes not on code but on handling the exceptions. The same applies to websites — an open catalogue is scraped quickly, a site with a login and bot protection takes longer and costs more.

How long it takes

A simple single-purpose script takes one or two days. A scraper that runs on a schedule takes three to five. A bot with a database and enquiry logic takes about a week. The deadline is named before the work starts, as a separate line in the estimate, and if it changes we say so straight away rather than on delivery day.

What you get at the end

Not “it works on our computer”. You get the source code, launch instructions in plain language and every credential. The script is yours: if it needs changing in a year, any developer can do it, and you will not pay twice for the same thing.

And if the source website changes

Plainly: a scraper is tied to the structure of the page, and when that structure is rebuilt it stops finding the fields it needs. That is not a defect but the nature of the task — every scraper behaves this way. Small fixes after delivery are free; a serious rebuild for a new structure is quoted by volume and agreed in advance.

Where to start

Describe the one job you do by hand most often. We will tell you what can realistically be automated in it, what is better left alone and what it would cost — before any work starts and with no obligation on your side.