Running Competitor Reports

In this lesson we will begin running various types of competitor reports. It should be noted that, while running these reports, each of the recommended tools will be providing additional data points that will be used in the Key Performance Indicators analysis. Therefore, it is recommended that you go through the rest of this lesson gathering information for one specific competitor, so that you can see how to best streamline the information gathering process for all the data points most efficiently.

It is also important to understand that there are many tools available that provide data estimates. The purpose of these lessons is not to go into depth on every tool available, but rather to give you a proven step-by-step approach for gathering valuable data about your competitors. However, it is also likely that the data is not 100% accurate. If you look at the Google analytics data for a website, and then compare that to the data provided by the many competitor analysis tools available online, you will surely find discrepancies in the information. All of these tools aggregate data from a variety of sources and make some assumptions based on statistical averages to come up with information such as traffic estimates, ad spend, etc. However, while the lens for each report may be skewed, at least it is similarly skewed for all of your competitors. Therefore, these tools do provide a good basis for comparative analysis.


This is a ranking report that will give you some amazing data about each keyword that a competitor is ranking for within the top 20 positions in Google organic SERPs. The data includes the keyword, the Google ranking position, Search Volume for each keyword, the URL of the page that is ranking, and the percentage of organic search traffic that is coming from that keyword.

How to get this data: There are numerous reporting tools that provide this type of information. Some are better than others. This lesson is not meant to provide a an in-depth comparison of the available tools, but rather to give you an easy path for getting the information that you need and making sense of that information. Therefore, the recommended tool to use in conjunction with this lesson is SEMrush.com SEMrush.com is a paid research tool with several membership levels starting at: $69.95/mo. This is a very useful tool and is highly recommended.

Additional Keyword Rankings Research Tools:

What it means: This will tell you what keywords your competitor is having success with, how important each keyword is to their overall strategy, and which pages you should evaluate on their site to learn more about how and why they are finding success with that keyword.


Step-By-Step Instructions The following step-by-step sequence will help you use SEMrush.com to quickly sort the organic ranking report for any competitor and turn it into a highly actionable tool. Follow these 10 simple steps.


Step 1 Go to SEMrush.com and log-in (a Pro account is recommended). Enter your competitors website into the search bar at the top of the page. In this example we are looking at fool.com which is the Motley Fool website, one of the most successful financial investment advice sites online. Once you enter your competitors website the tool will go to work and deliver a fascinating and insightful report dashboard. From here you can see a high level overview of the organic search traffic that is being driven from Google organic search results, an overview of the AdWords ad campaigns the competitor is running, some information on back links pointing to the competitors site and some information on display advertising. For our purposes we are going to concentrate on the organic search results.



Step 2 In the navigation column on the left side of the browser window, underneath the “Organic Research” section, click on “Positions”. Then, click on “Export” it is the button on the right side below the graph.



Step 3 When the “Export Data” lightbox pops up, click on the Excel option and the report will begin downloading.



Step 4 Open up the newly downloaded Excel file and you will see a bunch of great data. Next we will turn all of this data into an easy to understand and use ranking report.



Step 5 In this step we are going to delete all but 4 columns from this report. In the image below we have highlighted the columns to be deleted in blue. Once you are done the only columns that should be left are: Keyword, Position, Search Volume and URL.



Step 6 Now that we have narrowed the data down to our 4 important columns: Keyword, Position, Search Volume and URL… we are going to sort the data. So select the entire sheet. This can be done by clicking the upper left hand corner. It’s the cell that is above row 1, and to the left of column A. Once the entire sheet is selected, go to Data then Sort. Note: These screen shots are taken in MS Office 2011 for Mac. If you have a PC or a different version of Excel, the sort option may be located in a different place… but the functionality is still the same.



Step 7 In the below screenshot you can see how we want to sort the information. First sort by URL from A to Z. You can add a second sorting criteria by clicking the “+” and then select to sort by Search Volume and from Largest to Smallest. Then click OK.



Step 8 Once the data has been sorted by the criteria in step 7, we are going to create a Pivot Table. Start by selecting columns A through D. You can do this by clicking on A and dragging your mouse over to column D. Next choose Data and then click Pivot table.



Step 9 Once you select columns A-D and click Pivot Table, a new tab will be created in the Excel workbook with your pivot table. Also the PivotTable builder will appear as a pop over. Make sure the the little box is checked next to each field name. Next, you can configure the 4 quadrants with each data type by dragging fields between each area. See the screenshot below and make sure that the information in your Pivot Table is set up the same.



Step 10 Once you have configured the fields of your Pivot Table, you can click out of the Pivot Table builder and see that the report is now showing you every keyword that each page of your competitors website is ranking for in Google. In column A you will see the URL of each page highlighted in green with a list of the keywords that the page is ranking for listed below it. Column B is showing you the average monthly search volume for each of the keywords. Column C is showing you the position where your competitor is ranking for each keyword in Google’s SERPs.



With this information you can get a great understanding of how successful each page on your competitors site is and what keywords are driving that success. This information can be used for keyword targeting when building content of your own and getting a sense for what level your own content must perform at and how much success you can achieve using similar strategies to what your competitor has already implemented. Just remember, if you want to compete, you will have to outperform your competitor in terms of quality content, user engagement and back links to your page. But this exercise will give you an incredible road map.




This is a report that provides all of the most essential information about your competitors AdWords campaigns, including: Keywords, Ad Title, Ad Description, Landing Page URL, Cost Per Click, Search Volume for each keyword, estimated clicks per day and estimated ad spend.

How to get this data: SEMrush.com is the tool used for creating step-by-step instructions and examples in this course. However, numerous other tools are available that provide similar data.

Additional Paid Advertising Research Tools:

What it means: If you have competitors that have been advertising for a long time, you can learn a tremendous amount of useful information about how their campaign is performing and save yourself a lot of wasted money by simply replicating what is obviously working for them. Conversely, this can keep you from wasting money when you see something is not working for your competitor.


Step-By-Step Instructions The following step-by-step sequence will help you use SEMrush.com to download and analyze your competitors entire AdWords campaigns. If your competitor has been spending significant amounts of money on AdWords for an extended period of time, this information could give you the benefit of any experience and refinement that your competitor has already gone through.


Step 1 In this example we will take a look at the AdWords campaign being run by a popular line of natural personal hygiene products called Toms of Maine. Begin by logging into SEMrush.com and entering your competitors root domain into the search box at the top of the page, click search and then click on "Overview" (right beneath the SEMrush logo). This will give you an overview of your competitors performance in Organic Search, Paid Search, Backlinks and Display Advertising. In this lesson we will be diving into the Paid Advertiisng.



Step 2 In the navigation bar on the left side of the browser window, click on "Positions". This will show you a complete report on all of the AdWords ad campaigns that your competitor is running. Next, click on "Export" on the right side below the graph.



Step 3 Once you click on "Export" choose to export the data as an Excel file.



Step 4 When you open the Excel file it will help if you first select row 1 by clicking on the #1 in the upper left of the spreadsheet. With row 1 selected choose Bold and a background color for the header row. Next, select cell A2 and then go to the "Window" drop down menu and select "Freeze Panes" (note: these instructions are based on MS Office 2011 for Mac, if you run a different version of Office the "freeze panes" option may be in a different location). The reason for this formatting is that it will make the workbook much easier to work with moving forward.



Step 5 You now have a complete download of your competitors AdWords account. Column A represents the Title for each Ad, Column B includes both lines of the Ad description (two lines of up to 35 characters each), Column C tells you the keyword being targeted by that ad, Column D tells you which position the Ad has been appearing in on average over the past month, Column F tells you the search volume for that keyword and Column G tells you the estimated Cost Per Click for that ad. Column H tells you the destination URL for that ad.



Step 6 If you click on "Ads Copies" in the left navigation, SEMrush will actually render each ad so that you can see how they appear in Google search results.



Step 7 If you would like to only see ads that contain certain keywords or phrases, or filter out ads that contain certain words, phrases or URLs you can click on "Filters" and select your desired filter criteria. Once you click "Apply" the entire AdWords campaign list will be narrowed down based on your selected criteria. This is an incredible tool that can give you a lot of help in setting up your own AdWords campaigns.



Now that you know how to use this powerful tool to analyze your competitors AdWords campaigns, it is important to remember that while copying the campaign of a very successful competitor can go a long way to helping you replicate their success... copying an ad campaign that is losing money will help you replicate those losses. So, it is always best to closely monitor your ad spend and never turn an ad campaign up to full throttle until you have tested it yourself and proven that the ads will produce good ROI for your business.


This is a report that will contain the domain names for every website that is linking to your competitors.

How to get this data: For our purposes in this course we highly recommend Ahrefs.com. While no backlink database can be guaranteed to find all links that exist online, Ahrefs.com has easily the largest and most complete backlink database. Other tools can find similar information, however when comparing the different tools it quickly becomes obvious that Ahrefs.com delivers the best results.

Additional Backlink Research Tools:

What it means: This report gives you a great set of websites that you may want to include in your link building strategy. Visit each site, conduct your own evaluation to ensure that it is not a low quality site, and then attempt to contact the webmaster for the site to build a relationship that will result in them linking to your site. This is time consuming, however if is one of the most powerful things you can do.


Step-By-Step Instructions The following step-by-step sequence will help you use Ahrefs.com to easily download a report of the domains that are linking to any competitor.


Step 1 Go to Ahrefs.com, enter your competitors domain into the search box and click “Search Links”.

competitor link reports step 1 in competitive link analysis


Step 2 Once you enter a competitors domain and click search you will see the back link report overview. There is a tremendous wealth of data available here. From this screen you can see a complete overview of the numbers behind all the different types of links pointing to your competitors website including counts of individual types of links. For our purposes here, click on the blue number next to “Referring Domains” to the left of the graph. In the image below the blue text box is pointing to the “Referring Domains” report link.

referring domains link reports step 2 in competitive link analysis


Step 3 This is the referring domains report. Click on “Export” on the right side just above the report.

download the referring domains link reports step 3 in competitive link analysis


Step 4 Choose the format for the exported file and select the number of domains that you want to download. By clicking on the orange button the report will contain a list of the top 1,000 most authoritative domains linking to your competitor. However, if you want to download all of the domains linking to your competitor you can click the dark blue “Start Export” button.

choose file format for the referring domains link reports step 4 in competitive link analysis


Step 5 Once you have downloaded and opened the report in Excel you should format the header row to make the text bold and give the background some color, then select cell A2 and freeze panes. Lastly, resize the columns so that you can easily see all of the data. Formatting the reports in this way will make them much easier to work with.

formatted referring domains link reports step 5 in competitive link analysis


The back linking domains report will give you a good high level overview of how many sites are linking to your competitors website as well as the type of sites and quality of sites that are linking to them.




This is a report that lists every single link pointing to your competitors website, as opposed to just listing the domains like the above report, this one will list every single individual link.

How to get this data: For our purposes in this course we highly recommend Ahrefs.com. While no backlink database can be guaranteed to find all links that exist online, Ahrefs.com has easily the largest and most complete backlink database. Other tools can find similar information, however when comparing the different tools it quickly becomes obvious that Ahrefs.com delivers the best results.

Additional Backlink Research Tools:

What it means: This report will allow you to go to the actual individual pages that are linking to your competitors and see how their links have been integrated. You should do this to help you propose the right type of relationship with the webmaster in an effort to get links back to your own site.


Step-By-Step Instructions The following step-by-step sequence will help you use Ahrefs.com to easily download a report of the every individual link that is linking to any competitor.


Step 1 Next we will download the “Total Backlinks” report so that you can quickly and easily see the individual pages that are linking to your competitor. This will better prepare you for reaching out to the domain owners when trying to get them to link to your site by letting you see the types of links and content strategies that your competitors have successfully used to get links. Go back to the main overview dashboard and click on the blue number next to “Total Backlinks”. It is pointed out in the image below.

download competitors total links reports step 1 in total back link analysis


Step 2 This is the “total back links” report. It is similar to the “linking domains” report. Click “Export”.

find competitors backlinks step 2 in total back link analysis


Step 3 Select your chosen file type and the number of links that you would like to download.

choose file type for competitors backlinks report step 3 in total back link analysis


Step 4 This is the unformatted link report. Adding some formatting will make the report much easier to work with. See then next screenshot image.

open competitors backlinks report step 4 in total back link analysis


Step 5 This is the total back links report once formatting has been added.

final formatted competitors backlinks report step 5 in total back link analysis


Once you have these reports, you should plan to evaluate the back links and reach out to any potentially valuable back link partners and attempt to get them to link back to your website.




This is a crawl that allows you to see your competitors entire website and page specific SEO information, laid out into a spreadsheet.

Tools to identify competitor URL structure.

  1. Screaming Frog SEO Spider This tool is great! It is a spider program similar to Googlebot, that will crawl and index a website and give you a lot of the most important, page specific SEO information for every page on the site. The free version will crawl up to 500 pages on the main site, but you can upgrade to the paid version to find and crawl more pages, including subdomains and secure pages that have been tagged “no index, no follow”. When doing competitive analysis, you definitely want this info. Download and try the free version, then upgrade at your discretion. Honestly, this tool is so awesome, it is well worth the 99 Euros per year for the full version.
  2. Ahrefs SEO Reports Sign-up for an account at Ahrefs.com a minimum of the Professional level will be necessary to complete this competitive analysis. The Screaming Frog SEO Spider does give you a bit more information (if you have the full version of that tool), but if you already have an Ahrefs.com account for back link analysis, this tool is acceptable because it does provide most of the same information that Screaming Frog provides. This is also a tool that we will be using a lot throughout these courses. So go ahead and sign up now.

What to do with the information:

  1. Both of the above tools export the information into an Excel spreadsheet. From there you can gain deep understanding into your competitors information structure, page specific SEO strategies and page specific link network.

  2. This can give you a plethora of ideas for your own site and an understanding of what it would take to replicate your competitors strategy. That information can be used to estimate time, money and resources the project will require.

Step-By-Step Instructions The following step-by-step sequence will help you use Screaming Frog SEO Spider to completely crawl any website, yours or your competitors, and gain a fairly detailed analysis of many of the important SEO elements on the page.


Step 1 Go to http://www.screamingfrog.co.uk/seo-spider and download the Screaming Frog SEO Spider. It is recommended that you upgrade so that you can adjust the settings to crawl more pages and deeper into a competitors site… but start by downloading and using the free trial. It allows you to crawl up to 500 pages of a competitors site and gives you a good idea of how the tool works.



Step 2 Open the Screaming Frog SEO Spider and enter a competitors URL into the box and click “Start”. For this lesson we will run a report on CallawayGolf.com, a leader in the golf industry.



Step 3 Monitor the progress bar and once the Spider is 100% complete with the site crawl, click the export button and save the file to the appropriate place on your computer.



Step 4 When you first open the downloaded report in Excel it will be completely unformatted. Here is how to format the report to make it easier to work with:

  1. Delete rows 1 and 2.
  2. Now highlight row 1 and make it Bold and choose a background color of your liking.
  3. Select cell A2 and freeze panes so that you will always be able to see the header row as you are working with the data.
  4. Adjust the width of each column by hovering over the center divide between the letters at the top of each column, the cursor will change form and allow you to click and drag the column separators to the desired width.


    5. Step 5 The final report should look like the image below.



    The following list explains what the information in each column of the report is, and what it means.

    • Address: The URL of each page crawled. This may include sub-domains as well as pages that are internal indexes. If you find the SEO spider is crawling pages that are completely outside of the intended site, then you need to adjust the crawl settings by going to Configure, Spider, and deselecting “Crawl All Subdomains”, “Crawl Outside of Start Folder” and “Follow External “nofollow”.
    • Content: This tells you the type of content it is. When sorting the site crawl it is often helpful to sort all of the data by content first and then separating different types of content into different tabs within the workbook.
    • Status Code: This is the HTTP status code of the page. Once you have sorted the data into different tabs based on content type, it is also helpful to sort the data into different tabs based on HTTP status code.
    • Title 1 This is the information that is contained in the Title tag in the header section of the source code for the page. There should only be one title on the page, however, if the page has both a title tag and a meta title tag, then the spider will pick up on that and display both title 1 and title 2.
    • Title 1 Length: This is the number of characters in the title. A general rule of thumb is that you want a title to be less than 57 characters, but that can change based on the width of the characters being used.
    • Title 1 Pixel Width: The pixel width limit for titles as they are displayed in Google search results is 512 pixels. If your title goes over this limit, Google will not display the whole title and will be more likely to display a page title that is automatically generated by their algorithm.
    • Meta Description 1: The fist meta description in the source code for the page. There should only be one, however if your code lists multiple descriptions the spider will display both descriptions 1 and 2 in different columns.
    • Meta Description 1 Length: This the character count of the description. The general rule of thumb is that you want to stay under 160 characters.
    • Meta Description 1 Pixel Width: Like with the page title, Google now uses a pixel width limit for descriptions. The limit is 923 pixels and is broken up over two lines of text. If you go over this pixel limit Google will not show the full description and is more likely to generate their own description for you automatically through their algorithm.
    • Meta Keyword 1: This is the content of the meta keywords tag on the page. It is widely believed that Google’s algorithm no longer uses this tag to benefit a page, however if you stuff too many keywords into this tag it could possibly have a negative impact on the page.
    • Meta Keyword 1 Length: This is the number of characters used in the keyword tag.
    • H1-1: This is the content of the H1 on the page. There should not be more than one H1 tag. However, if the spider picks up more than one H1 tag on any pages on the site, then both H1-1 and H1-2 will be displayed.
    • H1-Length: The character count of the H1.
    • H2-1: This is the content of the first H2 on the page. It is perfectly acceptable to use multiple H2’s on a page.
    • H2-1 Length: The character count of the H2.
    • H2-2: This is the content of the second H2 not the page. The SEO spider will only report on the first two H2 tags indexed on the page.
    • H2-2 Length: The character count of the H2.
    • Meta Robots 1: The contents of the Meta Robots tag.
    • Meta Refresh 1: If the page is using a Meta Refresh tag, it will be listed here. These can be a very dangerous tag and should not be used unless you really know what you are doing.
    • Canonical Link Element 1: Every page should have a Canonical link to provide clarity on which version of the page is intended to hold all page rank in case of duplicate content issues.
    • Size: The total size of all elements on the page.
    • Word Count: The word count on the page.
    • Level: The number of clicks it takes to get to the page from the homepage.
    • Inlinks: The number of on-site links pointing to this page.
    • Outlinks: The number of links pointing off of the page.
    • External Outlinks: The number of links pointing from this page to a page that is not located within the site.
    • Hash: This can be used to identify duplicate content. If you sort the entire data set by this column, any two Hash numbers that are an exact match indicate duplicate pages.



Obviously you need to know the URL of your competitor. However, the URL structure of their categories and subcategories, along with their subdomains and use of SSL can all tell you a lot about both the information architecture, and conversion funnels of their site.

This information can help in site architecture planning. By learning about what information competitors are offering on their websites and how that information is being organized, you can ensure that your own site is offering a greater depth and breadth of content and that the content is organized in the best way possible. This will also help ensure that you are not missing entire sections of content that your competitors are addressing.

Examples:
  1. The category / sub-category structure and the specific words and phrases within URL structures can tell you a lot about their SEO keyword strategy.

  2. Sites use sub-domains in interesting ways such as housing their blog, support, international / alternate language versions, ecommerce, and sometimes to feature client pages. Important sub-domains must be identified to conduct a proper competitive analysis.

Tools to identify competitor URL structure.

  1. The HTML sitemap on your competitors site. Most sites will have at least two types of sitemaps. An HTML sitemap will typically highlight the major sections of the site for visitors. This doesn’t typically allow you to see all of the pages on the site, and may leave out many types of pages that play a key role in your competitors total web strategy. But, it will show you the information architecture strategy that they are using for SEO content.

  2. XML Sitemap. Typically you can find the XML sitemap for a website at: http://website.com/sitemap.xml This is the sitemap that is designed to tell search engines which pages should be indexed, how often those pages are updated, and how important each page is to the overall strategy of the website.

  3. https://ahrefs.com/seo-reports/ Sign-up for an account at https://ahrefs.com/pricing-plans.php a minimum of the Professional level will be necessary to complete this competitive analysis. However, this will tell you much, much more about your website than the previous two strategies. This is also a tool that we will be using a lot throughout these courses. So go ahead and sign up now.

  4. http://www.screamingfrog.co.uk/ This tool is great. It is a spider program similar to Googlebot, that will crawl and index a website and give you a lot of the most important, page specific SEO information for every page on the site. The free version will crawl up to 500 pages on the main site, but you will need to paid version to find and crawl more pages, including subdomains and secure pages that have been tagged “no index, no follow”. When doing competitive analysis, you definitely want this info. Download and try the free version. Upgrade at your discretion.

  5. http://www.powermapper.com This is a program available for PCs and is one that can really help you get a very good sense for how each of your competitors have organized their site. PowerMapper will create a visual sitemap for any website. This will be very important and valuable for both competitor analysis and site architecture planning. Go to http://www.powermapper.com and download it.

What to do with the information:

  1. All of the above tools export the information into an Excel spreadsheet. From there you can gain deep understanding into your competitors information structure, page specific SEO strategies and page specific backlink profiles.

  2. This can give you a plethora of ideas for your own site and an understanding of what it would take to replicate your competitor’s strategy. That information can be used to estimate time, money and resources the project will require.



Now that we have these awesome reports...

Let's Figure Out What They Mean!


Next In the next lesson we will take a look at all of the Key Performance Indicators on the Competitor Data Sheet. It is important for you to understand what each of these data points really means and how these data points will begin to reveal the strategies your competitors are using to achieve their success.

We will learn to identify your competitors strengths (so that you can replicate those strategies) and weaknesses (so that you can outperform them).

So, let’s jump in and take a look at each of your competitors