Community Hosted Servers Compromised
This last week, 5 of the 8 servers that are loco hosted but Canonical sponsored, had to be shut down due to reports that they were actively attacking other machines. These servers were found to have a variety of problems including, but not limited to, missing security patches, FTP (not sftp, without SSL) was being used to access the machines, and no upgrades past breezy due to problems with the network cards and later kernels. Loco teams will be given a choice to: a. migrate to the Canonical data center, or b. stay on the hosted/outsourced servers. Each option has its good and bad points. Jono Bacon has therefore called for a meeting to discuss these issues. The meeting will be in IRC #ubuntu-locoteams on Tuesday, August 14, 2007 at 2:00PM UTC.
Slashdot readers note: these machines were not hosted by Canonical and were running an old unsupported version of Ubuntu.
－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－
Changes to LoCo Server Policy
Jono Bacon jono at ubuntu.com
Sat Aug 11 01:23:46 BST 2007
* Previous message: CD Set with all software
* Next message: Next steps restoring hosted loco team websites (meeting summary)
* Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi all,
Recently we have had some problems with regards to LoCo servers, and the
situation has become untenable. I spent some time this week chatting
with James Troup, also known as elmo, and he leads the Canonical
sysadmin team. He has written up a mail outlining the issues well and
summarising the change in policy. This is important to encourage great
services, but also secure services.
I would like to schedule an IRC meetin in #ubuntu-locoteams on Tuesday
14th August at 2pm UTC to discuss any of the issues covered here.
The email is to follow:
--------------
Hi,
What happened
-------------
On Monday evening (UK time) it was reported that one of the hosted
community servers that Canonical sponsors had been compromised. After
investigation, it became apparent that 5 of the 8 machines had been
compromised. Since it was reported that they were actively attacking
other machines (and because it's What You Do), the decision was taken to
shut the machines down.
On Tuesday morning we started the procedure of bringing these machines
up in a safe state so that we could recover data from them.
Unfortunately, this took far longer than we would have hoped or liked
due to a combination of having to use remote hands, arbitrary limits
imposed by those remote hands and (relative) lack of bandwidth to copy
data off site.
This process is still ongoing (though only one remain has yet to be
fully recovered - tiber).
How did this happen
-------------------
Unfortunately:
a) the servers, especially zambezi were running an incredible
amount of web software (over 15 packages[1] that we recognised)
and
of all the ones where it's trivial to determine a version, they
were without exception out-of-date and missing security patches.
An attacker could have gotten a shell through almost any of
these sites.
b) FTP (not sftp, without SSL) was being used to access the
machines, so an attacker (in the right place) could also have
gotten access by sniffing the clear-text passwords.
c) The servers have not been upgraded past breezy due to problems
with the network card and later kernels. This probably allowed
the attacker to gain root.
What happens next
-----------------
We're obviously working as fast as we can to restore services, however,
we need to make sure they won't immediately be compromised
again.
Our first thought (and as previously mentioned on the loco-contacts
mailing list) was to simply move all these services into the Canonical
data centre, which would solve (b) and (c) above. However, at the time,
we weren't aware of (a).
Unfortunately it's simply not possible for us to maintain that amount of
software in any sane or secure fashion. So we've changed plan
slightly and now plan to do two things:
Loco teams/services can choose to either:
(1) be migrated to the Canonical data centre. This comes with both
restrictions and benefits:
+ Better hardware and bandwidth.
+ Fulltime support from Canonical's sysadmin team including
software maintenance and integration into our existing backup
infrastructure.
- root access will not be available.
- Access by per-user SSH key only, limited number of accounts
per loco team / service.
- Can only support certain software (e.g. drupal, wordpress,
planet, moin, ...).
- No ability to run arbitrary CGIs.
(2) or stay on the hosted/outsourced servers.
However, assuming anyone chooses option (2), some things will have to
change with how we handle these servers. Specifically, Canonical will
continue to sponsor the servers but they will have to become entirely
community run, i.e.
+ Community admin team liaise with hosting company for reboots, etc.
+ Community have sole responsibility for all aspects of
administration of servers, including but not limited to day to
day sysadmin tasks, backups, security, upgrades, recovery if
compromised, etc.
+ Use of servers for loco team services only unless previously
agreed.
- Both the Community Council and Canonical have oversight on this
--
James
[1] art-web, gallery, drupal, phpmyadmin, wordpress, postnuke, phpbb,
smf, moodle, planet, aspseek, moin, taskfreak, cms made simple,
mediawiki, ...
----------------
So, if you have any questions, come to the meeting and discuss them
there where both James and I will be present.
Jono
--
Jono Bacon
Ubuntu Community Manager
jono(at)ubuntu(dot)com
www.ubuntu.com / www.jonobacon.org
* Previous message: CD Set with all software
* Next message: Next steps restoring hosted loco team websites (meeting summary)
* Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the loco-contacts mailing list
