Loading...

Posts in category operations

New SSL certificate

We have updated the SSL certificate and intermediate CAs again today.

Fingerprints:

  • SHA256: 39:B8:E5:9B:F1:39:92:33:99:EF:32:5A:9C:8E:14:58:42:63:83:F2:A2:2E:36:FE:33:E7:91:16:21:A8:F6:81
  • SHA1: B0:53:47:AB:EE:9F:A0:67:F8:9C:4A:CF:81:72:A2:A2:46:47:1F:32

New SSL certificate

The *.coderesort.com SSL certificate has been renewed.

Fingerprints:

  • SHA1: 1D:4A:4D:BD:07:E6:B0:7B:90:40:FA:1C:FE:D1:A1:12:9B:31:3E:08
  • MD5: DB:82:8A:4D:3D:FC:8A:CF:D0:65:E9:AD:A3:88:25:1D

Maintenance - OS Upgrades

We are scheduling OS upgrades and maintenance across our infrastructure this weekend.

Services will be unavailable from 23:30 CET Friday night, but we expect most things completed in 2-3 hours. However, we are keeping the maintenance window open across the weekend in case follow-up issues require reboots or service restarts.

Follow @CodeResort on Twitter for further status updates.

Update: Upgrades completed in 6 hours. All services fully operational again.

Scheduled downtime 2011-09-19 at 16:00-17:00

Our ISP is making some infrastructure changes, and have warned that we can expect something like ~20 minutes of downtime on Monday September 19th after 16:00.

No Internet Link

Our Internet link broke twice this weekend (all times are CET):

  1. On Saturday August 27th at 11:42 the link went down. We called for emergency help and the ISP arranged for someone to come in and check cables. Unplugging and replugging cables, and as if by magic the link became active again at 16:36. 4h54m downtime.
  1. On Sunday August 28th in the evening the link started to become unreliable. Lots of packets not making it through the link, effectively slowing the service to a crawl. At 23:45 the link died completely. Contacted ISP emergency help again, but as it was now nighttime there was no real hope of immediate help. We scheduled a new visit first thing Monday morning. They arrived as promised, changed various cables and plugs, and measured all cabling for control. Link reconnected at 09:57, logging 10h12m of downtime.

We are quite confident the problem was related to cabling, and our ISP is quite confident that current cabling is OK. We sincerely hope the issue is resolved for good.

Note: When the service becomes unavailable, our Twitter account is the only reliable source of information. Do follow us if you want reliable status messages when issues arise.

DNS servers not reachable

Provider of our domain and DNS services, NetworkSolutions.com, is experiencing repeated distributed denial-of-service attacks. The DNS servers that map www.coderesort.com to address 82.196.207.189 has at times not been available. We are sorry about this and any problems you may be experiencing, but it seems that the worst is now passed as the servers have started responding again.

If you are not already following us on Twitter, please follow us now:

Follow CodeResort on Twitter

Any issues will be reported and tracked on Twitter as they happen. If our service is unavailable for any reason, @CodeResort will be our only reliable channel for reaching you with the latest status.

Renewed SSL Certificate

We have just deployed a new SSL certificate as our old certificate was set to expire in the next month.

As Subversion clients often come without any pre-defined certificate authorities, each new certificate must be explicitly accepted. For our new certificate, the approval should contain information similar to this - particularly verify the fingerprint:

Error validating server certificate for 'https://www.coderesort.com:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: *.coderesort.com
 - Valid: from Mon, 06 Jun 2011 06:30:33 GMT until Wed, 06 Aug 2014 18:06:16 GMT
 - Issuer: GeoTrust, Inc., US
 - Fingerprint: 45:f4:c0:4b:e8:e6:1e:21:21:a5:68:c3:d7:18:c6:63:d6:26:62:3e
(R)eject, accept (t)emporarily or accept (p)ermanently?

Please accept it (p)ermanently to make Subversion client work as before.

Scheduled OS upgrade tonight at 23:50 CET

We will do a major refresh of the server OS tonight, so expect up to 1 hour of downtime as we upgrade and verify.

UPDATE: Recorded downtime from 23:51 CET to 00:18 CET for a total of 27 minutes. All systems operational again.

Scheduled OS upgrade tonight at 23:45 CET

We will perform OS upgrades tonight, starting at 23:45 CET. Expect site to be unavailable for +/- 30 minutes while we upgrade packages, reboot and verify.

UPDATE: All upgrades completed. 10 minutes of downtime recorded.

Disruptions to Service and Trouble Committing Changes

We have had some trouble with a "standby" (backup) slave this weekend.

We experienced a short power loss across our server infrastructure, and in the subsequent automatic reboots an offline "standby" (virtual) slave has also been mistakenly powered on - allowing it to receive direct traffic intended for the main server. When the standby-slave periodically synced from its master, any changes recieved by slave directly was cleaned out.

Your changes between Sunday January 31st ~04:00 CET and Monday February 1st 14:00 CET may be affected by this. Please check your project to see if anything is missing.

If you have committed new revisions to a repository, or updated your working copy with changes from others, your local working copy may well be out of sync. The safest option is to check out a new, clean working copy and then transfer any missing changes to the new working copy and commit them there. Please Contact Us if you are unable to resolve this easily, and we'll help you take corrective actions.

For any non-repository changes and registrations affected by the incident, the only real solution is to re-type and post the new/updated information.

We are really sorry for any inconvenience this may have caused you, and have now taken measures to prevent this situation from arising again.

Unexpected Downtime & New Internet Connection

We experienced an unexpected downtime this afternoon - roughly from 16:00 CET to 17:00 CET. The error occured when a new planned link was configured by our ISP before it was planned to be physically connected, making it impossible to reach our network.

After locating the error we also then connected the new fiber-based line. With the recent server upgrade and this new fiber-based Internet connection, the site should now have very good performance.

Sorry for any inconvenience the downtime may have caused.

Server Upgrade Friday Oct 9th

We will be upgrading our server infrastructure on Friday night, and this will likely involve some downtime as we migrate. The window for making the changes is Friday Oct 10th 21:00 CET to Saturday Oct 11th 06:00, so during this time the services may not always be available. Naturally we will try to keep the actual downtime to a minimum.

Update: The upgrade is completed and everything should be working normally. The site was offline for approx. 25 minutes during the upgrade time window. Hopefully you will all notice that the site is much more responsive now, and please contact us if you experience any problems.

New SSL Certificate

We have updated our SSL certificate. If you see a message similar to:

Error validating server certificate for 'https://www.coderesort.com:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: *.coderesort.com
 - Valid: from Fri, 13 Feb 2009 11:58:50 GMT until Mon, 04 Jul 2011 10:58:50 GMT
 - Issuer: Equifax Secure Inc., US
 - Fingerprint: ee:79:36:14:81:be:cb:d8:90:ca:80:3c:48:0d:32:0e:49:48:c5:e2
(R)eject, accept (t)emporarily or accept (p)ermanently?

... then please accept it.

It should not be an issue with browsers, but as Subversion clients usually come without any pre-approved Certificate Authorities, all certificates must be explicitly accepted.

Issue with sporadic hang is fixed

In recent weeks and months we have experienced an issue whereby the web projects become unresponsive - basically unable to serve your requests. During this time we have kept a very close watch to make sure we get it back online as soon as possible. Still, we know that some of you no doubt have been inconvenienced by these short downtimes.

The good news is: We have discovered and fixed the cause of the issue.

A bug in [[TOC]] wiki macro only appeared when requests where made to a page in a project that used the macro, and such a project would then permanently claim one database connection. It 'infected' a project on first view, but it did not get worse by itself. However, at random intervals pages with the macro was requested on other projects tying up new connections until at some point no further database connections could be established.

(Those that are technically inclined can read about the issue and see my fix at http://trac-hacks.org/changeset/4366)

Finding this bug has taken much longer than we had imagined. However, it has led us to review all the central parts of the code, and fixing and reworking some parts that were not optimal. Additionally, we have updated all server software to the latest and greatest versions. The service is now in very good shape.

We can only apologize, and thank you all for being patient with us trough this period.

E-mail Delays Today

There seems to have been an e-mail delay today with our ISP (Broadnet.no).

We have contacted them, and the have confirmed that the there was in fact a problem, and that the problem should now be corrected. However, they warned that some further delays can still be expected as the e-mail servers catch up on the backlog of outgoing e-mails during the coming hours.

Sorry for the inconvenience, and please contact us if the delay persists longer than can be reasonably tolerated!

Renewed SSL certificate

We have just renewed our SSL certificate, and the new certificate is operational as of now.

Depending on your browser, Subversion client, feed reader or other client accessing CodeResort services, you may receive a message along the lines of:

Error validating server certificate for 'https://www.coderesort.com:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: *.coderesort.com
 - Valid: from Tue, 03 Jun 2008 09:40:45 GMT until Mon, 04 Jul 2011 09:40:45 GMT
 - Issuer: Equifax Secure Inc., US
 - Fingerprint: 5a:d9:93:3e:a5:66:12:ea:8f:44:fe:04:41:f4:5c:b9:42:9e:6d:b2
(R)eject, accept (t)emporarily or accept (p)ermanently?

Please accept the new certificate permanently, and everything should work as normal.