Jasu
When someone searches for «SEO audit automation with AI«, they usually expect a list of expensive tools or a magic agent that solves everything on its own. The reality is that this process could be automated quite a few years ago (minus a human eye that also makes some decisions or customises certain aspects). Once the audit is systematised, the next step is usually to automate SEO reports to turn recurring data into periodic decisions.
Once the audit workflow is under control, the next step is usually to automate SEO reporting to turn recurring data into monitoring, alerts and actionable decisions.
In this article I show you a specific workflow for semi-automating a technical SEO audit using Python, Screaming Frog, Google Drive and Looker Studio. There is no AI at the core of the process, and that is precisely the point.
At the end, you will see some ideas for using AI in the process.
An automated audit should not stop at technical errors: it can feed predictive SEO with AI and better SEO strategies with AI if it cross-references crawl data, Search Console, SERPs and business opportunities.
What «automating» an SEO audit really means
Automating does not mean nobody does anything. It means removing repetitive tasks such as opening Screaming Frog manually, exporting twelve tabs of data one by one, copying and pasting into Excel, uploading it to Drive, updating the dashboard…
Those steps take up a fair amount of time per project that could be spent on other steps.
Semi-automation targets the stretch where we gather all the information about the domain and put it into a presentation document. The analysis and conclusions are still our own, although today artificial intelligence could help us with this step.
The complete workflow: Screaming Frog + Python + Drive + Looker Studio
The workflow we are going to look at has three linked steps. Each one solves a specific problem and, together, they turn a morning of manual work into something that runs while you focus on other tasks.
Step 1 – Python interface to launch Screaming Frog from the command line
Screaming Frog has a CLI mode (command line interface). It lets you launch crawls from the terminal without opening the graphical interface, passing the URL, crawl depth, exports to generate and destination folder as arguments.
The problem is that the syntax is long, unintuitive and error-prone if you type it by hand every time. To solve that, I created this open source script: github.com/queseo/Screaming-frog-with-Python-Interface.
What it does is provide an interactive command-line interface: it asks you for the site URL and the configuration options, then launches the crawl with the correct command. You do not have to remember flags or paths. You type what you want to audit and the script takes care of the rest.
Depending on the configuration, Screaming Frog generates between 8 and 15 CSV files with different data: URLs, redirects, images, headings, hreflang, sitemaps, structured data…
I should say that the script is old and has not been updated for the new versions.
Screaming Frog does now let you upload all the files to Drive, but it uploads (or used to upload) the files separately, and that can be a bit of a problem if you have to keep hunting for the specific sheet each time.
Step 2 – Merge all the exports into a single Excel file
Fifteen loose CSV files are not a report. The same script collects all those files from the exports folder and combines them into a single Excel file with one sheet for each type of data.
The result is a structured file ready to upload to Google Drive.
This step only requires pandas and openpyxl. The code iterates over the CSV files, normalises them minimally (upper-case columns, basic data types) and writes them into the Excel file with one sheet per file.
Step 3 – Looker Studio as a dashboard connected to Drive
With the Excel file in Drive, the last step is visualisation. Looker Studio (formerly Data Studio) can connect to Google Sheets files or Drive files as a data source.
The strategy here is to create a Looker Studio template with each sheet of the Excel file connected to a different data source. One sheet for URLs with 4xx errors, another for redirects, another for duplicate titles, another for Core Web Vitals… Each block of the dashboard pulls data from the corresponding sheet.
Once the template is set up, it is reused for every project. You only have to point the data source at the Excel file of the client in question. The whole dashboard updates itself. There is nothing to build from scratch for each new audit, but if there are custom fields, they need to be adjusted manually.
You do not need AI to have an automated audit process
It is tempting to think there is no automation without AI, but this is neither the first nor the last process to be automated before AI.
Tools such as Oncrawl, Semrush, Ahrefs and Se-Ranking also have their own audit tools. They have limitations when it comes to customisation, but they include APIs to extract the information and connect it to tools that display the data and cross-reference it with our audit.
For example, exporting Semrush project data is somewhat more complex than what we can see in their API documentation (I do not know whether they have updated it, but if not, here I explain how to export Semrush project data via the API).
If we combine all this with Python (which has loads of ready-made resources for SEO) and Google’s APIs, we have access to a huge amount of information without needing AI to work.
AI makes sense in the steps that involve interpretation, not in those that involve mechanical execution. Moving data from one place to another, renaming columns, generating an Excel file with X sheets: programming languages already do that.
Automating is not a synonym for «with AI». Very often it is a synonym for «with code». And code, in this case, is more reliable, cheaper and easier to audit than any agent.
That said, there are parts of the workflow where AI can add real value. And there are quite a few of them.
How to improve this workflow with AI
Once you have clean, centralised data, AI has much more context to work with. Here are some ideas for specific ways to bring it into the workflow without everything depending on it.
Smart prioritisation of technical issues
Not all errors are equal, and when we optimise a website we come across lots of things that need fixing.
An agent with access to crawl data and traffic metrics (GSC, GA4) can automatically calculate a priority score for each issue and suggest where to start optimising and what to prioritise, giving us a plan for the tasks to be done.
Keyword research
Classifying keyword research takes time and, although AI cannot automate the whole process, it can help us with certain classifications as long as they are small.
This is where it helps to lean on automation tools such as make or n8n, although we can also build our own system directly with code or agents.
Improved client presentations
Another step we could take: instead of creating the presentation from Looker Studio, these days that Excel file could be handed to an AI so it creates a presentation directly with all the information.
It occurs to me that a well-prepared skill in Claude could work very well in this process.
Automating audits with make / n8n / AI agents
This could also be done, but setting up the process would take a lot of time, bearing in mind that we are already starting from very good tools on the market that keep growing.
But if we want more customised options adapted to our processes, it is an interesting step, mainly for larger agencies that have their own system and want to invest in developing, improving and optimising it.
Frequently asked questions
Do I need programming skills to use this workflow?
A basic level of Python is enough to install the dependencies and run the script.
Is Screaming Frog CLI available on every licence?
CLI mode requires a paid Screaming Frog licence. The free version has no access to the command line and does not allow unlimited crawls.
Can I use this workflow for recurring client audits?
Yes, that is exactly what it is designed for, but the script has not been updated for the latest versions of Screaming Frog SEO Spider.
What happens if the site has a lot of URLs and the Excel file becomes very heavy?
Looker Studio handles files of up to several thousand rows per sheet well. For very large sites (hundreds of thousands of URLs), the best option would be to connect directly to BigQuery or a database instead of going through Excel, and to have a more customised system for visualising the information.
Can AI make mistakes when analysing the crawl data?
Yes. LLMs can misinterpret numerical data or prioritise badly if the context is not well structured. That is why, in the proposed workflow, AI acts as a supporting analysis layer, not as the main executor. The SEO should always validate the outputs before delivering them.
For agencies that sell recurring diagnostics, a white-label SEO tool can help present audits under their own brand, as long as human judgement reviews the priorities.
Automating audits can save time, but it needs review and clear priorities. I cover the hiring side in how to find an SEO consultant with AI.
Automation can speed up checks, although commercial priority should be reviewed by hand. I put it into practice on the SEO audit for Gandia page.
Technical SEO consultant in Valencia for more than 16 years. Speaker and organiser at WordCamp Valencia and author on the Semrush and Publisuites blogs. More about me
