Status of NLANR Caches

June 2, 2005

NCAR has decided that they can no longer host bo1.us.ircache.net.

June 2, 2005

bo1.us.ircache.net is down again, this time due to a "critical UPS component failure."

May 18, 2005

bo1.us.ircache.net is down for the next four days due to scheduled maintenance at the hosting site.

May 13, 2005

sj.us.ircache.net is having serious hardware issues, most likely with the hard drives. It will likely be unavailable for a few days.

January 14, 2005

pb.us.ircache.net has been replaced with a newer box. The PB cache is the last of the original DEC alphaserver boxes purchased at the start of the IRCache project, back in 1995. Long live PB!

July 15, 2004

The SV cache was taken offline tonight because the hosting site recieved a dshield abuse complaint about a SINGLE ICMP packet from joe@a1poweruser.com. As a result of this abuse report, the cache was offline for more than 12 hours and the hosting site is now requiring us to disable the useful ICMP-based network measurement feature.

June 18, 2004

This morning the peering router used to host the SJ cache went down hard due to failed hardware. It looks like it could be down at least through the weekend and until we can get a card replacement from the vendor.

May 21, 2004

The SJ cache experienced a power outage this morning and will not fully boot. I will be visiting the site during the next few days to fix it.

Feb 18, 2004

Changed our access-extract.pl script to ignore /squid-internal- requests. This should make the hit ratio calculations a little more sensible..

Dec 13, 2003

UCSD, the host for sd.us.ircache.net, is under a DDOS attack, apparently for publishing a report on another DDOS attack.

Sept 12, 2003

Local staff disconnected the RTP cache because the "County of Riverside Network Security Team" reported a SYN flodd attack.

Sept 10, 2003

The NY cache went offline (hung) and now cannot find its operating system. A new box is on the way to the site.

May 28, 2003

The NY cache has been fixed.

May 21, 2003

The NY cache became unreachable tonight. I asked the on-site staff to power cycle the box, which helped only a little. We are still working to get the box fully running again.

May 13, 2003

The SV cache is offline because the hosting location decided it was using too much bandwith.

We are trying to find a new location/provider for the cache. Stay tuned.

February 6, 2003

The New York cache is now online: ny.us.ircache.net.

The old SV cache died and the replacement has been installed and put online. SV now has twice as much disk space and memory as before.

February 2, 2003

The cron job that distributes new password files was broken for a few days. Anyone who registered recently may have found that they could not access the caches. It has been fixed now.

January 29, 2003

sv.us.ircache.net became unreachable today. Initially it seemed like a network problem, but now appears to be a system problem. The on-site support folks have been asked to investigate.

January 28, 2003

A new cache, ny.us.ircache.net, will be online soon.

October 31, 2002

startap.us.ircache.net has been taken out of service.

October 9, 2002

startap.us.ircache.net will be taken out of service soon. Genuity has been donating the network connection to the STARTAP cache, but now they are going to start charging significant amounts of money.

We will be looking for a new home for the startap cache. If you can provide one, please let me know.

September 3, 2002

startap.us.ircache.net is going to be down for a couple of days because the hosting facility is short on rack space and they need to make room for equipment for a Very Important Demo.

June 14, 2002

pa.us.ircache.net is being relocated to a new facility and should be back online by monday.

June 12, 2002

bo2.us.ircache.net became unreachable from some locations today when the FRGP folks installed a new Juniper router. They know about the problem but are unwilling to fix it for fear of breaking something else.

February 22, 2002

pa.us.ircache.net became unreachable this afternoon. The Palo Alto area experienced a major power outage.

February 5, 2002

uc.us.ircache.net became unreachable at approx. 17:30 UTC. The UIUC folks said they had an unplanned power outage and they will send someone to look at the machine.

January 16, 2002

pb.us.ircache.net became unreachable at approx. 15:30 UTC. I've asked the PSC folks to investigate, but they have not replied.

uc.us.ircache.net has been unreachable for much of the day due to a botched remote OS upgrade.

startap.us.ircache.net is experiencing horrible packet latency all day:


 1  lsd6509.sl.startap.net (206.220.241.254)  0.313 ms  0.257 ms  0.209 ms
 2  m10-lsd6509.startap.net (206.220.240.154)  0.290 ms  0.316 ms  0.261 ms
 3  m5-m10.startap.net (206.220.240.105)  1.404 ms  1.504 ms  0.879 ms
 4  na165.nap.net (207.227.0.165)  3043.210 ms  3231.692 ms  3517.361 ms
 5  p2-0.chcgil1-cr1.bbnplanet.net (4.24.8.130)  3223.325 ms  3334.820 ms  3148.322 ms
December 17, 2001

sj.us.ircache.net is now running on new hardware, thanks to the folks at Worldcom/MAE-West!

October 17, 2001

rtp.us.ircache.net has been upgraded to a MUCH nicer box. This one has three CPUs and 2GB of RAM. Its also connected to a UPS, so I think it shall be much more reliable.

June 4, 2001

rtp.us.ircache.net is down today. NCREN staff are on vacation or at home, and cannot look at the box until tomorrow morning.

March 16, 2001

We switched bo1.us.ircache.net to an Intel-based PC system. The old DEC Alpha is being decomissioned, partly because the hosting site doesn't want to have FDDI hosts in their network any more.

Feb 1, 2001

STARTAP is being relocated today. It is being moved back to the STARTAP facility, which should provide better connectivity. Its IP address is changing as well.

Jan 23, 2001

PB became unreachable at 05:38 EST. The site is having a scheduled outage.

Jan 21, 2001
RTP was down today because of a power outage at the site.

Dec 30, 2000
RTP became unreachable at 12:00 PDT. NCREN staff will not be back at work to reset it until Tuesday, Jan 2.

Nov 21, 2000
A replacement startap.us.ircache.net is back online, although its temporarily in a lab rather than at the exchange point. Once the machine proves to be stable, STARTAP staff will relocate it to the other facility.

Nov 21, 2000
sv.us.ircache.net is back. The problem was that there was not enough free disk space for 'savecore'. From now on all DEC caches have SAVECORE_FLAGS set to "M" so the system will boot multiuser mode regardless.

Nov 17, 2000
sv.us.ircache.net is still down. Replacement of the FDDI card had no effect. Still no link light, still unpingable. Reconfirmed that the cable was good by plugging it into another FDDI device.

Nov 16, 2000
sv.us.ircache.net became unreachable last night at approx 21:00 PST. Fix-West folks think the FDDI card is faulty. I have fedexed a replacement card to the site.

May 31, 2000
startap.us.ircache.net crashed early this morning and will not boot up now. There is nobody on the site who can fix it yet.

March 24, 2000
We have a new cache online, named startap.us.ircache.net. It is located at the STARTAP internation exchange point, near Chicago, IL. Connectivity and transit is provided by BBN/GTE.

February 11, 2000
sj.us.ircache.net is down, probably until around 09:00 PST. I was fiddling with the network config and did something that made it unresponsive.

January 31, 2000
lj.us.ircache.net is being directed to sd.us.ircache.net via a DNS CNAME entry until we can get the old HP-UX box working again.

January 30, 2000
lj.us.ircache.net is down until Monday when someone will be in to power-cycle it.

December 1, 1999
We attempted to upgrade the system disk for rtp.us.ircache.net, but the new disk has media errors. We'll put the old disk back for now and try the upgrade again tomorrow.

September 16, 1999
rtp.us.ircache.net has failed with some serious hard disk errors. We hope to have it somewhat functional again within 24 hours.

August 23, 1999
squid.nlanr.net became a DNS round-robin entry. This causes some problems for people that configure their browser or cache with squid.nlanr.net as the hostname. You should use lj.us.ircache.net intead.

August 18, 1999
RTP was powered off this morning due to a air conditioning failure.

August 8, 1999
RTP failed early this morning for an unknown reason. The staff at NCREN have been contacted, but apparently they don't work on weekends.

July 30, 1999
Our Web/FTP server was down for a few hours during an operating system upgrade.

July 29, 1999
Lost a disk on uc.us.ircache.net. Cache size is down to about 18 GB.

February 18, 1999
We are renaming our caches to use the ircache.net domain. The new names are as follows:
	pb.us.ircache.net
	uc.us.ircache.net
	bo.us.ircache.net
	sv.us.ircache.net
	sd.us.ircache.net
	sj.us.ircache.net
	pa.us.ircache.net
	rtp.us.ircache.net

February 4, 1999
Lost a disk on bo2.cache.nlanr.net. Cache size is down to about 18 GB.

December 18, 1998
sv.cache.nlanr.net crashed some time during the night, and we have been unable to reach someone at the NASA facility to reboot it.

December 14, 1998
lj.cache.nlanr.net and sd.cache.nlanr.net will be offline for a few hours today while the machines are physically relocated.

November 16, 1998
bo1.cache.nlanr.net seems stable. The DNS has been updated so that bo resolves to both bo1 and bo2.

November 5, 1998
bo1.cache.nlanr.net is now back in service. However, we will leave the DNS unchanged for now, so that bo resolves only to bo2.

October 17, 1998
bo1.cache.nlanr.net is down with an apparently serious hardware problem. We will change the DNS to return only the IP address for bo2.cache.nlanr.net until the problem can be resolved.

October 2, 1998
rtp.cache.nlanr.net failed because of an overheated disk. We have sent a replacement to this location and hope to have it online by October 7.
PS -- Don't EVER buy 10,000 PRM disks!! Especially from companies which start with 'S'.

October 1, 1998
Disabling SSL requests helped, but we still have significant abuse of lj.cache.nlanr.net. Therefore, we have changed our open-access policy for that cache. If you want to use it, please just send us a note.

September 24, 1998
We have been forced to deny all SSL requests on all NLANR caches due to repeated abuse. Numerous sites relay SSL connections through our caches attempting to find valid, stolen credit card numbers.

August 29, 1998
rtp.cache.nlanr.net became unreachable. We have contacted the on-site staff at that location to reboot the machine.

July 27, 1998
All caches had poor performance due to a bug in 1.2.beta23 (second patch set). After the nightly reconfigure (SIGHUP), the dnsserver processes were being closed and all new requests were blocked on DNS lookups

July 27, 1998
The old sd.cache.nlanr.net DEC alpha has been replaced with a new PentiumII with 512MB RAM and 36GB of disk.

July 2, 1998
The IP addresses for sd.cache.nlanr.net and tirana.nlanr.net have been changed. The new addresses are:
sd.cache.nlanr.net      192.172.226.145
lj.cache.nlanr.net      192.172.226.146

May 5, 1998
We have installed a second SCSI controller in bo1.cache.nlanr.net.

April 23, 1998
uc.cache.nlanr.net is now fully reachable again. Apparently it was only cut-off from the vBNS.

April 22, 1998
We lost connectivity to uc.cache.nlanr.net at approximately 22:43 UTC. Apparently, it seems to be a router failure at NCSA and we are waiting for them to fix it.

April 7, 1998
sd.cache.nlanr.net failed at 11:00 AM. Still at 512 MB RAM.

April 6, 1998
We are attempting to return sd.cache.nlanr.net to 512 MB.

April 5, 1998
A disk failure on our Web and FTP server, ircache.nlanr.net, caused an outage of almost 24 hours. We apologize for the inconvenience. A new disk and operating system have been installed. If you notice any odd behaviour, please bring it to our attention.

April 2, 1998
All caches have been converted over to unicast ICP, because (a) the vBNS multicast has not been working for many days now, and (b) we are preparing to compare ICP with another next-hop selection technique, the results of which are to be presented at this summer's Web caching workshop.

March 31, 1998
sd.cache.nlanr.net is back, but with only 256 MB memory. It will run with a smaller cache size for the time being.

March 28, 1998
sd.cache.nlanr.net has failed for mysterious reasons. We have used a DNS CNAME record to temporarily direct sd traffic to lj.cache.nlanr.net.

March 25, 1998
We have turned off transit (default) routing for the MAE West Web cache router (sj.cache.nlanr.net). This reduces the number of clients which can reach the cache, and the number of origin servers which the cache can reach. Please see our MAE West NAP prototype page for more information.

March 23, 1998
Fixed a 1.2.beta17 bug which caused only ICP_MISS replies to be returned.

March 20, 1998
We are running Squid-1.2.beta17 on all our caches. Because of swap file format changes, all current cache contents will be lost. It will take a few days for the caches to fill up and reach their former hit ratio levels.

March 6, 1998
We have changed the access control scheme in an attempt to eliminate some improperly configured caches. In recent weeks we have noticed a number of cache clients sending us constant streams of ICP queries, all of which were DENIED. These sites are either running old, buggy versions of Squid, or non-Squid caches. Because neither cache in the relationship complains too much, this incorrect configuration continues, and potentially millions of useless ICP messages may be sent across the Internet.

Our solution to this is to allow ALL ICP queries, but still deny HTTP requests from sites which are not in our access lists. We use the deny_info feature to show the user a customized error message. This message explains why their request was denied and recommends they contact their cache administrator.

Hopefully, because we no longer have access controls on ICP queries, this change will have another nice side effect. We expect our CPU usage to go down a little, or perhaps we will be able to handle a higher rate ICP queries.

February 25, 1998
We are installing a second cache at the Boulder location. The old bo.cache.nlanr.net will be renamed to bo1.cache.nlanr.net and the new cache will be (cleverly) named bo2.cache.nlanr.net. The name bo.cache.nlanr.net will return addresses for both caches, for round-robin load balancing.

November 19, 1997
pb.cache.nlanr.net has been successfully upgraded to 512MB RAM! Finally!!!

October 23, 1997
Digital Equipment has given us a bigger caching machine for use at their Palo Alto Exchange! This is a screaming fast Digital AlphaStation 500/500 with 512MB of RAM and 60GB of disk. We have changed the DNS entry pa.cache.nlanr.net to point at the new machine. Also note that on this machine we will use Squid's default ports of 3128 and 3130.

October 10, 1997
sd.cache.nlanr.net has been successfully upgraded to 512MB RAM! Finally!

October 8, 1997
it.cache.nlanr.net has been suddenly and permanently disconnected. This machine was located at the Cornell Theory Center in Ithaca New York. Unfortunately, CTC is being forced to take less of a role in NLANR activities, so we cannot keep the cache there.

Instead, this cache machine is being relocated to MAE-WEST, in San Francisco California. Thats good news for West coast users, but not so good those on or near the U.S. East coast. We hope to also have a cache located at MAE-EAST in the near future.

August 22, 1997
Upgraded caches to Squid-1.1.16 which has a new closest-only option to the cache_host line. This option causes Squid to forward only ``CLOSEST_PARENT_MISS'' requests and never ``FIRST_PARENT_MISS'' requests. In other words, we will not forward the request unless the peer returns some ICP SRC_RTT data, and that peer has the lowest RTT to the origin server. In addition to causing each cache to handle requests for the origin servers closest to them, this feature also cuts down on the number of ERR_DNS_FAIL failures because the peer won't return any SRC_RTT data if it can't map the hostname to an IP address.

August 20, 1997
sv.cache.nlanr.net has been successfully upgraded to 512MB RAM. pb and sd are still at 256MB.

We are bringing a new cache online: pa.cache.nlanr.net is located in Digital's Palo Alto Exchange facility. Note, this cache will be running on port 8080. To use, add this to your squid.conf:

	cache_host pa.cache.nlanr.net parent 8080 8082

August 7, 1997
The cache registration service has not been working properly since sd.cache.nlanr.net was shut down two weeks ago. The announcement software has been installed on tirana and the registration database should begin to repopulate shortly.

August 6, 1997
The configuration file of sv.cache.nlanr.net was incorrectly changed on August 1 because we believed its memory upgrade was successful. When the cache filled up on August 5th and 6th, the squid process began swapping excessively, causing very poor service times.

July 25, 1997
it.cache.nlanr.net was successfully upgraded to 512 MB RAM.

uc.cache.nlanr.net was successfully upgraded to 512 MB RAM.

The upgrade of pb.cache.nlanr.net was NOT successful. It is still running with 256 MB RAM.

July 24, 1997
sd.cache.nlanr.net is behaving badly. It has suddenly crashed at least three times. Digital finally tested all the SIMMs and found many of them were faulty. This problem has already cost thousands of dollars, so we have decided to leave the machine off for now. Once again sd has been CNAME'd to tirana.nlanr.net.

July 23, 1997
sd.cache.nlanr.net is back in service (with 512 MB RAM) at 1930 GMT. The DNS entry has been restored but we expect it will take a few days for this to fully propogate.

July 18, 1997
sd.cache.nlanr.net has been down since approx 1900 GMT. Difficulties were encountered while attempting to upgrade the RAM. It is still down as of 0230 GMT. It looks like sd may be unavailable for quite some time, so sd.cache.nlanr.net has been CNAME'd to tirana.nlanr.net.

July 17, 1997
Memory upgrades arrived from Digital. bo.cache.nlanr.net was successfully upgraded to 512MB RAM. With the new memory, bo has been configured with
	cache_mem	32
	cache_swap      20000
	cache_swap_low  75
	cache_swap_high 95
Check the vital statistics to watch the cache fill up with more objects.

We'll do our best to keep this page current. If you think we're slacking off, send a reminder to Duane.


$Id: status.html,v 1.103 2006/08/22 23:36:15 wessels Exp $