Building AI for a startup providing content optimization

Search engines favor pages that provide a thorough overview of a topic that includes all fundamental subtopics, answer questions, and move user closer to satisfying their initial query. Nearly 75% of users never go past the first page of a search engine. Therefore, SEO is pivotal in determining the rank of websites. The client wanted to ease the process of data extraction from their client’s websites in order to improve their search performance. Algoscale leveraged Topic modeling, a complex form of AI to provide a solution that standardized their process and helped optimize their time.

The Client

Headquartered in Boston, U.S, the firm was founded in 2013. The client is a content marketing firm that uses AI to accelerate content planning, creation, and optimization. The client is a finalist for the 2018 Red Herring and Top 100 North America Award.

The Challenge

Search algorithms are getting progressively smart. Search engines engage models that measure the topical horizon of a page and not just the keywords. The client wanted a suggestion on all the best suitable and relevant topics that were to be covered in the content for improving their SEO ranking to draw the attention of more targeted customer segments.

A huge amount of data had to be cleaned and organized, and duplications were to be removed which led to increased complexity. Also, the data was to be extracted from diversified sources and open-ended domains.

The Solution

Algoscale team suggested building a web crawler that would standardize the process of data extraction from varied sources. Earlier the client needed custom crawlers for each new website which was inefficient. The basic idea was to standardize a crawler that could extract the data from different URLs and further the data would be cleaned, organized, and stored using SOLR. Algoscale used Scrapy, a web-crawling framework, which was independent of the format of the website and content, happened to provide a more accurate solution resulting in saving time for QA (Quality Assurance) analysis. The data extracted were junk-free and the quality test gave 90% accuracy.

Topic modeling, an established technique used to extract valuable topics from a corpus of data was used to determine specific keywords according to the needs, suggesting all the suitable and relevant topics that were to be covered in the content. MySQL database was set up to track the status of the crawler i.e., it gets updated automatically in real-time with the progress of the crawler for the website and as well as for avoiding any type of repetition. Thereby, improving the SEO ranking by optimizing content around user intent to draw the attention of more targeted clients.

Benefits

Algoscale’s solution helped the client in meeting deadlines because of the standardization of crawlers. Further, we increased the accuracy by 10-20% in extracting targeted topics thereby guiding the client to create world-class content which in turn drove more traffic to the website and earned links back to on-site content.