In this paper the web is analyzed as a graph aggregated by host and pay-level domain (PLD). The web graph datasets, publicly available, have been released by the Common Crawl Foundation(http://commoncrawl.org ) and are based on a web crawl performed during the period May-June-July 2017. The host graph has ~ 1.3 billion nodes and ~ 5.3 billion arcs. The PLD graph has ~ 91 million nodes and ~ 1.1 billion arcs. We study the distributions of degree and sizes of strongly/weakly connected components (SCC/WCC) focusing on power laws detection using statistical methods. The statistical plausibility of the power law model is compared with that of several alternative distributions. While there is no evidence of power law tails on host level, they emerge on PLD aggregation for indegree, SCC and WCC size distributions. Finally, we analyze distance-related features by studying the cumulative distributions of the shortest path lengths, and give an estimation of the diameters of the graphs. © Springer Nature Switzerland AG 2019.

Analysis of the web graph aggregated by host and pay-level domain

Funel, A.
2019-01-01

Abstract

In this paper the web is analyzed as a graph aggregated by host and pay-level domain (PLD). The web graph datasets, publicly available, have been released by the Common Crawl Foundation(http://commoncrawl.org ) and are based on a web crawl performed during the period May-June-July 2017. The host graph has ~ 1.3 billion nodes and ~ 5.3 billion arcs. The PLD graph has ~ 91 million nodes and ~ 1.1 billion arcs. We study the distributions of degree and sizes of strongly/weakly connected components (SCC/WCC) focusing on power laws detection using statistical methods. The statistical plausibility of the power law model is compared with that of several alternative distributions. While there is no evidence of power law tails on host level, they emerge on PLD aggregation for indegree, SCC and WCC size distributions. Finally, we analyze distance-related features by studying the cumulative distributions of the shortest path lengths, and give an estimation of the diameters of the graphs. © Springer Nature Switzerland AG 2019.
2019
9783030054137
Web graph;Structural network properties;Network analysis;Network models
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.12079/5858
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
social impact