HTTP 500 - squid vs java.net.URL Source

January 4, 2011 Markdown source

1---
2title: 'HTTP 500 - squid vs java.net.URL'
3date: '2011-01-04'
4published_at: '2011-01-04T16:21:00.004+11:00'
5tags: ['development', 'java', 'squid', 'tomcat']
6author: 'Gavin Jackson'
7excerpt: 'Ran into an interesting bug today that I thought I should share with the interwebs. We have a web application that supports RESTful http requests to interact with the system. Seems to work fine using...'
8updated_at: '2011-01-04T16:37:40.989+11:00'
9legacy_url: 'http://www.gavinj.net/2011/01/http-500-squid-vs-javaneturl.html'
10---
11
12Ran into an interesting bug today that I thought I should share with the interwebs. We have a web application that supports RESTful http requests to interact with the system.
13
14Seems to work fine using browsers/wget etc. But when trying to use java.net.URL (as used by several JUnit tests), I kept getting an http 500 (internal server) error message.
15
16Tomcat logs showed no interaction with the web server, indicating that it must be something wrong with the Squid reverse proxy server (which we have set up to cache outbound web traffic). SQUID LOGS
17
18```
19From java test harness (java.net.URL)
201294109122.110   1175 127.0.0.1 TCP_MISS/500 280 GET http://localhost:3128/geoserver/rest/gazetteer/latlon/-35.0,145.3 - FIRST_UP_PARENT/myAccel -
211294109146.216      0 127.0.0.1 TCP_NEGATIVE_HIT/500 306 GET http://localhost:3128/geoserver/rest/gazetteer/latlon/-35.0,145.3 - NONE/- -
221294109191.214      0 127.0.0.1 TCP_NEGATIVE_HIT/500 306 GET http://localhost:3128/geoserver/rest/gazetteer/latlon/-35.0,145.3 - NONE/- -
23...
24```
25
26Always comes back with 500 error code.
27
28From browser:
29
30```
311294109466.962   1226 127.0.0.1 TCP_MISS/200 698 GET http://localhost:3128/geoserver/rest/gazetteer/latlon/-35.0,145.3 - FIRST_UP_PARENT/myAccel application/xml
32```
33
34Note that the error code is 200 (not 500). 200 == HTTP OK 500 == HTTP Internal Server Error Subsequent harness requests come back ok (once it is in the cache). WHAT IS GOING OVER THE WIRE? So the question is, how do the http GET requests differ? Wireshark dump (from harness):
35
36```
37GET /gazetteer/latlon/-35.0,145.4 HTTP/1.1
38User-Agent: Java/1.6.0_22
39Host: spatial-dev.ala.org.auAccept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
40Connection: keep-alive
41HTTP/1.1 500 Internal Server Error
42Date: Tue, 04 Jan 2011 02:58:29 GMT
43Server: Apache-Coyote/1.1
44Age: 129
45Content-Length: 0
46X-Cache: HIT from ala-devmaps.vm.csiro.auX-Cache-Lookup: HIT from ala-devmaps.vm.csiro.au:3128Via: 1.1 ala-devmaps.vm.csiro.au:3128 (squid/2.7.STABLE5)
47Connection: close
48Content-Type: text/plain
49```
50
51Wireshark dump (from browser):
52
53```
54GET /gazetteer/latlon/-35.0,145.5 HTTP/1.1
55Host: spatial-dev.ala.org.auUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6
56Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
57Accept-Language: en-us,en;q=0.5
58Accept-Encoding: gzip,deflate
59Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
60Keep-Alive: 115
61Connection: keep-alive
62Cookie: __utma=100819395.1000872318.1288999346.1292997025.1293056871.6; __utmz=100819395.1288999346.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
63HTTP/1.1 200 OK
64Date: Tue, 04 Jan 2011 03:04:26 GMT
65Server: Noelios-Restlet-Engine/1.0..8
66Expires: Mon, 04 Jan 2021 03:04:26 GMT
67Content-Encoding: gzip
68Content-Type: application/xml;charset=ISO-8859-1
69Content-Length: 286
70X-Cache: MISS from ala-devmaps.vm.csiro.auX-Cache-Lookup: MISS from ala-devmaps.vm.csiro.au:3128Via: 1.1 ala-devmaps.vm.csiro.au:3128 (squid/2.7.STABLE5)
71Keep-Alive: timeout=15, max=100
72Connection: Keep-Alive
73.............N.0.E.~E....W...q?.].$....N;y-2..B....,..k..mr.f...Kv:(m....}....S......4T@.$..4.7.e..R....b>OK..M...B...De#.b.?.....Ph.7kt..o
74...).).....g.8 ..~.G..D..;m[..z...9...a4/...........
75!..........K-..`...\dkB/[...l.....uKKb.U...........(?...s_g...19.j..j..MW{\...........xnt....
76```
77
78Note that there are additional fields set in the GET request, after a bit more investigation, this is the GET request header that you need to set for this to work properly: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 So the following code works (note that if you drop the gzip,deflate option, you don't need the GZIPInputStream stuff):
79
80```
81...
82           URL url = new URL("http://spatial.ala.org.au/gazetteer/search?lon=148.866⪫=-36.195");
83           URLConnection connection = url.openConnection();
84           connection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
85           connection.setRequestProperty("Accept-Encoding", "gzip,deflate");
86           connection.setDoInput(true);
87           InputStream inStream = connection.getInputStream();
88           GZIPInputStream gis = new GZIPInputStream(inStream);
89           BufferedReader input =
90           new BufferedReader(new InputStreamReader(gis));
91           String line = "";
92           while ((line = input.readLine()) != null)
93           System.out.println(line);
94...
95```
96
97
98

---
title: 'HTTP 500 - squid vs java.net.URL'
date: '2011-01-04'
published_at: '2011-01-04T16:21:00.004+11:00'
tags: ['development', 'java', 'squid', 'tomcat']
author: 'Gavin Jackson'
excerpt: 'Ran into an interesting bug today that I thought I should share with the interwebs. We have a web application that supports RESTful http requests to interact with the system. Seems to work fine using...'
updated_at: '2011-01-04T16:37:40.989+11:00'
legacy_url: 'http://www.gavinj.net/2011/01/http-500-squid-vs-javaneturl.html'
---

Ran into an interesting bug today that I thought I should share with the interwebs. We have a web application that supports RESTful http requests to interact with the system.

Seems to work fine using browsers/wget etc. But when trying to use java.net.URL (as used by several JUnit tests), I kept getting an http 500 (internal server) error message.

Tomcat logs showed no interaction with the web server, indicating that it must be something wrong with the Squid reverse proxy server (which we have set up to cache outbound web traffic). SQUID LOGS

```
From java test harness (java.net.URL)
1294109122.110   1175 127.0.0.1 TCP_MISS/500 280 GET http://localhost:3128/geoserver/rest/gazetteer/latlon/-35.0,145.3 - FIRST_UP_PARENT/myAccel -
1294109146.216      0 127.0.0.1 TCP_NEGATIVE_HIT/500 306 GET http://localhost:3128/geoserver/rest/gazetteer/latlon/-35.0,145.3 - NONE/- -
1294109191.214      0 127.0.0.1 TCP_NEGATIVE_HIT/500 306 GET http://localhost:3128/geoserver/rest/gazetteer/latlon/-35.0,145.3 - NONE/- -
...
```

Always comes back with 500 error code.

From browser:

```
1294109466.962   1226 127.0.0.1 TCP_MISS/200 698 GET http://localhost:3128/geoserver/rest/gazetteer/latlon/-35.0,145.3 - FIRST_UP_PARENT/myAccel application/xml
```

Note that the error code is 200 (not 500). 200 == HTTP OK 500 == HTTP Internal Server Error Subsequent harness requests come back ok (once it is in the cache). WHAT IS GOING OVER THE WIRE? So the question is, how do the http GET requests differ? Wireshark dump (from harness):

```
GET /gazetteer/latlon/-35.0,145.4 HTTP/1.1
User-Agent: Java/1.6.0_22
Host: spatial-dev.ala.org.auAccept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
HTTP/1.1 500 Internal Server Error
Date: Tue, 04 Jan 2011 02:58:29 GMT
Server: Apache-Coyote/1.1
Age: 129
Content-Length: 0
X-Cache: HIT from ala-devmaps.vm.csiro.auX-Cache-Lookup: HIT from ala-devmaps.vm.csiro.au:3128Via: 1.1 ala-devmaps.vm.csiro.au:3128 (squid/2.7.STABLE5)
Connection: close
Content-Type: text/plain
```

Wireshark dump (from browser):

```
GET /gazetteer/latlon/-35.0,145.5 HTTP/1.1
Host: spatial-dev.ala.org.auUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cookie: __utma=100819395.1000872318.1288999346.1292997025.1293056871.6; __utmz=100819395.1288999346.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
HTTP/1.1 200 OK
Date: Tue, 04 Jan 2011 03:04:26 GMT
Server: Noelios-Restlet-Engine/1.0..8
Expires: Mon, 04 Jan 2021 03:04:26 GMT
Content-Encoding: gzip
Content-Type: application/xml;charset=ISO-8859-1
Content-Length: 286
X-Cache: MISS from ala-devmaps.vm.csiro.auX-Cache-Lookup: MISS from ala-devmaps.vm.csiro.au:3128Via: 1.1 ala-devmaps.vm.csiro.au:3128 (squid/2.7.STABLE5)
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
.............N.0.E.~E....W...q?.].$....N;y-2..B....,..k..mr.f...Kv:(m....}....S......4T@.$..4.7.e..R....b>OK..M...B...De#.b.?.....Ph.7kt..o
...).).....g.8 ..~.G..D..;m[..z...9...a4/...........
!..........K-..`...\dkB/[...l.....uKKb.U...........(?...s_g...19.j..j..MW{\...........xnt....
```

Note that there are additional fields set in the GET request, after a bit more investigation, this is the GET request header that you need to set for this to work properly: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 So the following code works (note that if you drop the gzip,deflate option, you don't need the GZIPInputStream stuff):

```
...
           URL url = new URL("http://spatial.ala.org.au/gazetteer/search?lon=148.866⪫=-36.195");
           URLConnection connection = url.openConnection();
           connection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
           connection.setRequestProperty("Accept-Encoding", "gzip,deflate");
           connection.setDoInput(true);
           InputStream inStream = connection.getInputStream();
           GZIPInputStream gis = new GZIPInputStream(inStream);
           BufferedReader input =
           new BufferedReader(new InputStreamReader(gis));
           String line = "";
           while ((line = input.readLine()) != null)
           System.out.println(line);
...
```