Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
Hardware

Coolest Cluster Ever 279

sw155kn1f3 writes "Scientists at Los Alamos National Laboratory built a cheap (less than $1k per unit) 294-unit Beowulf claster dedicated to run astrophysics calculations. According to their website it's 85th fastest computer in the world. Seems cool and promising as it made with cheap components and off the shelf hardware."
This discussion has been archived. No new comments can be posted.

Coolest Cluster Ever

Comments Filter:
  • if only machine-making like this became mainstream. what would we use the computing power for? SETI? i say we can never have enough power. *insert tim the toolman joke*
  • Imagine... (Score:5, Funny)

    by Anonymous Coward on Monday November 25, 2002 @01:30AM (#4749118)
    Imagine a single unit of these...
  • by Anonymous Coward
    beowulf cluster of these running off the energy of these radioactive trees in the last story.

    -)

  • Astrophysics calculations eh? So basically it's an ugly R2-D2...great. Really fantastic.
  • Obligatory (Score:4, Funny)

    by BSDevil ( 301159 ) on Monday November 25, 2002 @01:32AM (#4749131) Journal
    Imagine a Beowulf cluster of these...Beo...shit - I knew this joke would have to end at some point.
    • Oddly enough, I was just going to make the exact same post. Thank you for sparing me. Freenet IS the hope of the future
  • Uh oh (Score:3, Insightful)

    by Eric(b0mb)Dennis ( 629047 ) on Monday November 25, 2002 @01:36AM (#4749147)
    Astrophysics.... 85th largest in the world... Call the department of homeland security, this could be terrorism! Yeah, okay.. it was stupid :P
    • Re:Uh oh (Score:3, Funny)

      by MoThugz ( 560556 )
      Yeah, it's stupid... as you can see from the article itself, everyone involved in that project has a @*.gov email.

      Sheesh...
    • Re:Uh oh (Score:2, Funny)

      by Decimal ( 154606 )
      Astrophysics.... 85th largest in the world... Call the department of homeland security, this could be terrorism!

      We should actually be worried about the eithty-fourth fastest computer in the world... Saddam Hussein has built it out of a warehouse of imported PlayStation 2s.
  • by Max Romantschuk ( 132276 ) <max@romantschuk.fi> on Monday November 25, 2002 @01:36AM (#4749148) Homepage
    They didn't even use a rack mount solution, they used regular Shuttle XPC SS51G [shuttleonline.com] Mini-PCs

    I thought Shuttles Mini-PCs were cool before but this really resets the scale... Now where is the HOWTO for this thing? ;)
    • Since these are the guys that did most of the original work on LinuxBIOS [lanl.gov], does that mean that they have ported linuxbios to the Shuttle XPC SS51G motherboard? I would love a P4 2.5GHz PC that boots into linux in about 3 seconds!
    • by ryochiji ( 453715 ) on Monday November 25, 2002 @04:20AM (#4749770) Homepage
      > They didn't even use a rack mount solution

      I was thinking that too when I saw the picture. I mean, how are they securing those boxes? The way it looks, I certainly wouldn't want to be standing near that rack when an earthquake hits...

      But then, imagine: Cause of death: crushed by Beowulf cluster. That's a geek's dream come true!

      • "
        But then, imagine: Cause of death: crushed by Beowulf cluster. That's a geek's dream come true!" - ryochiji
        Yeah, the kind of dream you wake from out of breath and with your heart racing. I, for one, am not yearning for a violent, ignoble death. But for those of you that are, it's probably easy to arrange.
    • They didn't even use a rack mount solution, they used regular Shuttle XPC SS51G Mini-PCs

      The question has to be... why? I've never understood why so many people build clusters with essentially desktop PCs. Haven't these people ever heard of the 1U rack mount case? Yes, it's a slightly more expensive inital outlay. But surely the cost savings in terms of floorspace and power/cooling for such a large volume would outweigh that in no time flat? Plus the Shuttle, like most other desktop cases, don't have the option of hot swappable drives. With the number of machines in this sort of cluster, drive failure is a major problem. The ability to just pull the failed drive out of the front to replace it would be a huge win.

      • Actually, you should be careful about the benefits of making the servers more and more dense! 1U servers are a pain to deal with for cables, and keeping the things cool. If you have the space, there is almost no real benefit to fitting the equipment into as tight a footprint as possible.
    • they used regular Shuttle XPC SS51G [shuttleonline.com] Mini-PCs
      Well, duh!! Of course they used shuttles - it's for a space simulation!
      http://space-simulator.lanl.gov/ [lanl.gov] ;-)
  • by autopr0n ( 534291 ) on Monday November 25, 2002 @01:36AM (#4749152) Homepage Journal
    I don't really think you can really quantify coolness in general, but I fail to see how the fact that this thing is cheap makes it all that cool.

    Perhaps if it was going to run simulations of ultra-low temperature physics... get it? haha. I kill me.
  • Maybe it's just me, but the "85th" fastest computer in the world just doesn't do anything for me. I mean, sure, it's cool and all, but... Really... 85th?
    • C'mon....

      For $294,000US, this has to impress you.

      All of the top 10 fastest computers in the world are multi-multi-million dollar machines. This is a breakthrough because it represents another milestone in bringing supercomputing accessible.

      $264k one day, $100k the next. I sincerely hope that soon small-to-medium enterprises can own supercomputers. With all the low budget physics stuff going on at Universities around the world, cheap supercomputing can only be a good thing.

      • by BWJones ( 18351 ) on Monday November 25, 2002 @03:12AM (#4749571) Homepage Journal
        I sincerely hope that soon small-to-medium enterprises can own supercomputers. With all the low budget physics stuff going on at Universities around the world, cheap supercomputing can only be a good thing.

        Actually they can with software like that from Dauger Research [daugerresearch.com], Project Appleseed [ucla.edu] and Wolfram Research [wolfram.com] with gridMathematica [wolfram.com]

        The cool thing here is that this code can be run on all of the desktop computers that already occupy companies and universities world wide allowing for easy access to supercomputer level computational speed (for those problems that can be attacked using parallel computation of course) using the same computers normally used for productivity.

        Very cool.

      • What kind of environmental impact does a cluster like this have over a super computer. Hardware wise it is cheaper to build but what does it cost to power it, cool it, and house it?
  • by TWX_the_Linux_Zealot ( 227666 ) on Monday November 25, 2002 @01:37AM (#4749163) Journal
    It would be possible for a group of people, not necessarily a small group, but not necessarily huge, either to repeat this. 100 people, each with $3,000, could do it. The group would need to find some space to house the thing, and would probably have to do it in a climate where it could be relatively naturally cooled, which definitely rules out Phoenix. The computer would then be one of the fastest machines in the world.

    Granted, I don't know what the hell they'd do with the computer, but it would be kind of cool to be on the list.
    • Granted, I don't know what the hell they'd do with the computer

      kick your opponent's butts in [insert favorite fps here]... ???
    • actually. if these people are not after money, then this would be a great computer to work out simulations for social dynamics, alternatice economic system theory, etc.

      The point being, if you are creaticve, you could find quite a few interesting things to do.

      • Good point. All the supercomputers in the world today are being used by big money programes. I can't wait until this is on my desktop (or on a network I can use).

        There are thousands of complex systems that could be modeled that might provide fasinating insights. What happens when you run the Game of Life for 10^100 generations? How about compiling the Linux kernel in a genetic algorithim? Who knows?

    • Or.... (Score:5, Funny)

      by raehl ( 609729 ) <(moc.oohay) (ta) (113lhear)> on Monday November 25, 2002 @01:57AM (#4749275) Homepage
      They could spend their $3k on something actually needed by 100 people thinking about spending $3k to share a spot on a fastest computers list.

      Like a prosititute.

      You go to your high school reunion, what's, more impressive, the "Hugh Hefner" 100 $3k prostitutes that come with you, or the "Bill Gates" story about the 300 1k computers in your mom's basement?
    • "It would be possible for a group of people, not necessarily a small group, but not necessarily huge, either to repeat this. 100 people, each with $3,000, could do it. The group would need to find some space to house the thing, and would probably have to do it in a climate where it could be relatively naturally cooled, which definitely rules out Phoenix."

      Who'd pay the insurance and electric bills?

      "Granted, I don't know what the hell they'd do with the computer..."

      They'd spend another amount of cash on a OC-48 connection to the internet and then offer webhosting to website owners who think they might be next in line of getting linked in a story at slashdot.com ...
    • Granted, I don't know what the hell they'd do with the computer, but it would be kind of cool to be on the list.

      Duh! Doom III of course!

  • And, just for the record, I would like to say that I know these guys and they're very nice, too. They work in my Dad's building. Nifty. Go T-Division!
  • Why build one when they could more easily have just leased the processing power from Intel?
  • Now if there was ever an advert for Shuttle XPC systems, the image does that just nicely.

    If I only had 1 Shuttle XPC, that would be great. I suppose Shuttle ought to add this site to their news section. Hopefully their web server runs off these systems and a fat internet pipe... just to test the /. load.

  • by SystematicPsycho ( 456042 ) on Monday November 25, 2002 @01:44AM (#4749201)
    Take a look at this room A 1000-Pentium Beowulf-Style Cluster Computer [genetic-programming.com] half way down the page.
  • Yes, but... (Score:1, Offtopic)

    Imagine a radioactive [slashdot.org] beowulf cluster of these!
  • Bull (Score:1, Redundant)

    by CanadaDave ( 544515 )
    I don't know how they can claim to be the cheapest Beowulf cluster at $1k/unit. That is $1k UDS per unit I assume, which is $1500 CAD (actually more than that due to bad exchange). Okay, let's say $1550 CAD. Well with that I could buy a pretty sweet computer, actually a REALLY sweet computer, considering it doesn't need a monitor, or a fancy video card. I mean, we're just talking mobo, CPU, fan, RAM, network card, case. I could easily make a small cluster. In fact I was using small cluster at work in the summer. It has about 3 or 4 to begin with, and I bet the cost about $1500 CAD, but I'm pretty confident they were less. They were just 1.2 GHz Athlons. We ended up dismantling it (reducing it to only 2 units!) but still, it is cheap. I just don't see how they make that claim.
    • Shit, am I on crack? They never claimed to have the cheapest Beowulf cluster... I'm sorry, I really must have been on crack. I read the website then came back to ./, started to write, got distracted with something else, and then must have forgotten the exact details. Ignore post, and mod down, please.
  • This computer can be found along with the other 499 fastest supercomputers [top500.org] on the to500.org [top500.org] website. Looks like Los Alamos has 2 out of the top 3 fastest.

    I'd like to say USC has a machine in the top 100, but we don't. :(

  • Perhaps... (Score:2, Funny)

    by wd123 ( 209211 )
    Instead of astrophysics work they should use it to find all their radioactive trees [slashdot.org].
  • Whoah.. (Score:3, Funny)

    by Anonvmous Coward ( 589068 ) on Monday November 25, 2002 @01:48AM (#4749222)
    "According to their website it's 85th fastest computer in the world. Seems cool and promising as it made with cheap components and off the shelf hardware."

    I'm guessing the story submitters ran out of anti-MS ammo tonight. Heh.

    It's satire, laugh.
    • Re:Whoah.. (Score:4, Funny)

      by enneff ( 135842 ) on Monday November 25, 2002 @02:46AM (#4749459) Homepage
      I thought satire was supposed to be funny.
      • "I thought satire was supposed to be funny."

        That misperception disappeared when SNL inexplicably rose to popularity.
    • So let's have some flame here...

      When it will be the time M$ reaches the top 500 on supercomputing? Linux has been for long there. And it is getting nearer and nearer the first places. However, till now, Redmond couldn't manage to gather even a humble supercomputer made of crappy Windows. What is strangeas there are libraries for parallel computing on Windows.

      So it seems that Windows is not ready for the bleeding edge... And no one knows when it will be...
      • Good point; MCSEs always boast of Windows' clustering ability (needed for reliability more than anything,) although IIRC it's more high-availability clustering than parallel computational clustering. That said, as highly as MS touts Windows Datacenter server, it's surprising (well, OK not really) that a system running Datacenter Server hasn't even cracked the top 500.
      • When xbox live goes on line and MS secretly moves forward with phase 2, harvesting 5 to 10% of cpu cycles of their subscribers, which they then sell.

        Yeah, I know it's not true, but it definately seems like it might have been a good idea.
  • Reminds me of.... (Score:2, Insightful)

    by Drakonite ( 523948 )
    Reminds me of when a computer company did the same thing... I think it was HP but I could be wrong.

    Although if I recall correctly they ended up quite a bit higher on list.

  • by dethl ( 626353 ) on Monday November 25, 2002 @01:49AM (#4749238)
    Last modified: November 15, 2002

    We are the last to know about it :P
  • Hmm... (Score:5, Funny)

    by lvdrproject ( 626577 ) on Monday November 25, 2002 @01:52AM (#4749255) Homepage
    1. Create Beowulf cluster
    2. ???
    3. SLASHDOT!

    Laugh, it's a joke.

    :Lav

  • As much as I am a fan of cheap 'beowulf' custers, there is a certain issue which troubles me. In this case, the cluster in question cost less than 300,000$, a healthy sum, but much less than a large cray or sgi server would cost. Such clusters can be used for an array of activities, such as nucular bomb tests (one of the driving forces in pushing supercomputer technology is nucular weapons), or cracking encryption.

    Supercomputers are controlled by USA export laws, but powerful beowulf clusters can be made by anyone with a reasonable amount of money and knowlage. Since the software is free and of the shelf components can be used... wouldn't it be possible for terrorists to use open-source software to create their own supercomputers to test nucular weapons, crack American law and millitary encryption, ect... ?

    I believe this 'beowulf' techonology, as great as it is, could be possible dangerous to American interests. It is my hope that this software will soon be controlled by the American millitary and not be spread for free because I fear for the safety of my family and country (bless them both) if terrorists have access to supercomputing technology.
    • Ladies and gentlemen, we have a replacement for Jon Katz...
    • Are you a former lawmaker, or otherwise mentally deficient?

      • The bag is here, the cat is long gone. Taking beauwulf source code off the net is like taking pee out of a pool. (blatantly stolen from News Radio)
      • The physics behind nuclear weapons is well understood and widely published. It took a small town full of geniuses to figure it out the first time, (and determine that it's even possible) but no longer. They also designed a Uranium gun-type weapon and a Plutonium implosion-type weapon. The latter is muchmuch more complicated. Designing a gun-type Uranium weapon is trivial, and anly slightlymore difficult if you're restricted to pen, paper, and sliderule. (Granted I'm an MIT student, so I'm probably overestimating the amount of brain power available to a large terroist organization.) The difficulty lies in the tremendous amount of effort/electrical energy/sapce/time/money required to perform isotope seperation/smuggling without getting caught before you have the chance to detonate the weapon.
      • Cracking bulk ciphers is basically infinately parallelizable and does not require much communication bandwitdth at all. Beauwulf clusters don't gain you much. You'd get better performance running your cracking programs over V2OS, FreeDOS, or some other ultra-lightweight OS.
      • Bin Laden et al. are very smart. Sure they have some grandios long-term plans about getting the US out of Saudi Arabia, creatig an Islamic super-state, and acquiring a fission bomb. However, they are very smart and efficient. They use low-infrastructure, reliable attacks for the most part. Planes filled with fuel, boats filled with conventional explosives, and dirty bombs are all very simple and can be potentially acquired by a single autonomous cell. I'd be much more worried about poison gas attacks and coordinated massively distributed firebomb attacks. These are the kind of low-tech high-casualty attacks at thier disposal right now.
  • by serutan ( 259622 ) <snoopdoug@@@geekazon...com> on Monday November 25, 2002 @02:08AM (#4749326) Homepage
    The photo alone is worth surfing over to the article. As Socrates once said, "what a rack!"

    But now that they've got the 85th fastest computer, what will they have to do to maintain that coveted position? I imagine the people who are running 86th are rushing out to buy more nodes. My own computer is the world's 27,385,422nd fastest, and I'm battling like crazy to get to 27,385,421. :-)
  • Astrophysics? Pfft! How bout a LAN party? Does 10 billion fps on UT2003 sound good?
  • Since the headline reminds me of the Comic Book Guy, I will dedicate this post to him. Besides, if this cluster, supercomputer, beo thingy is only 85th fastest in the world, you would think that they would use it to figure out the Comic Book Guy's rating scale (or something of equal importance) as opposed to some physics mumbo jumbo.

    An excerpt:

    [BABF01] Treehouse of Horror X: Desperately Xeeking Xena [snpp.com]

    (The Collector, slowly, strikes a dramatic pose)

    Collector(CBG): Lucite hardening ... must end life in classic Lorne Greene pose from "Battlestar Galactica." Best ... death ... ever!

  • Yes! (Score:2, Funny)

    by Konster ( 252488 )
    It must be running the website, too. It hasn't been /.ed yet.
  • On a side note, what's with the cheapo racks they have em on? This whole thing looks to be one seismic wiggle away from disaster.
    • That Nexel/Metro type of racking is a lot sturdier than it looks in the photo. All that open space lets air circulate and reduces the surface area for dust to build. Shelving like that beats finding out that you have just the right amount of rack space for the your new toy but the available holes don't line up.
  • by dr.Flake ( 601029 ) on Monday November 25, 2002 @03:06AM (#4749546)
    I really like the picture on the sites frontpage.
    i can imagine the small size of the Shuttles being an advantage, not to mention the "coolness" factor looking at it. (i assume the "cool" in the intro refers to emotion and not teperature!!)

    But getting computation done cheap is no longer the challenge. It's getting the data from one node to the other. They still need "custom" expensive equipment for this.

    I see they use 3com gigabit ethernet. having this 300+ gigabit switch capability is not "cheap".

    Until one can buy this kind of networking equipment for really cheap, we shouldn't mention things like "of the shelve Beowulf super computer in the top 100".
    • by tconnors ( 91126 ) on Monday November 25, 2002 @05:43AM (#4750039) Homepage Journal
      I really like the picture on the sites frontpage.
      i can imagine the small size of the Shuttles being an advantage, not to mention the "coolness" factor looking at it. (i assume the "cool" in the intro refers to emotion and not teperature!!)

      But getting computation done cheap is no longer the challenge. It's getting the data from one node to the other. They still need "custom" expensive equipment for this.

      I see they use 3com gigabit ethernet. having this 300+ gigabit switch capability is not "cheap".

      Until one can buy this kind of networking equipment for really cheap, we shouldn't mention things like "of the shelve Beowulf super computer in the top 100".


      Up until 3 hours ago, we [swin.edu.au] said we were the only machine in the top500 dedicated solely to astrophysics. Now we are one of 2 :)

      But we use 180 processors, @ 2.15 GHz, and get about 0.90 Gflops/processor, whereas they get about 0.88 gflops/proc, getting us in the top 180th.

      The difference being they have faster memory, and we have a big badass switch. They have two switches, with something like only 10 gigabit between the switches! We have 250gbit within our one switch. a third of our nodes have 2 gigs RAM, and we also have room for upgrade to more nodes on our switch, and they don't. So, in the words of Nelson "Hee haw!" :)

      When they say $1000 per proc, they are not factoriing in their two switches. This will bring the price up to about $500,000, unless someone is donating a switch or 2 :)

      We have about the same cost ratio - something like 250,000 .au dollars for all our procs (and maybe switch - I don't know the details) - half the performance, half the cost.
  • I'd ask how hot is the room. The heat pumps move the heat from the CP ... into the room.
  • by Xenolith ( 538304 ) on Monday November 25, 2002 @03:14AM (#4749577) Homepage
    Disclaimer: I have never built a cluster before.

    What is the advantage of having a hard drive in each node. Can't you boot each node off of a networked image and load the OS and whatever "work" into memory.

    thanks
    • Back in the days when I was working with LAVCs (Local Area VAX Clusters) over 10-Mbit Lans, diskless systems were possible but any disk I/O hit the bandwidth of the entire network. Pageing could murder the system. Also for larger clusters, simply loading the system on a number of machines took time as a mass storage server can only serve one system per request.

      Ok, modern LANs, especially this one is a lot faster, but you still don't want to burden your cluster communications bus with disk I/O requests.

      Anyway, that 80 gig Maxtor does not add much to the cost of the node.

      • Anyway, that 80 gig Maxtor does not add much to the cost of the node.

        It's not the price of a hard disk in a cluster node that causes the trouble, it's the fact that you now have one more power-drawing, heat-generating, moving part - that's more likely to fail than anything else in the system!
  • Seriously, assuming that you had the resources to build a large cluster, what would you do with it?

    And I'm hoping I won't get the obligatory "pr0n collection" jokes.
    • PVM-Pov is probably the first thing I'd put on it. Then I'd probably hack around with image processing experiments - and (of course) data mining my erotica library and turning my pr0n collection into a massively parallel screensaver...
  • The network switch is composed of a Foundry FastIron 1500 switch trunked to another FastIron 800 switch, which provides a total of 304 Gigabit Ethernet ports using the 16-port JetCore modules.

    What's the bandwidth of that trunk? Also, what's the capaity of the connections between each 16-port card and the backplane?

    Just curious... suppose all the units on a 16-port card have 1Gbps each, but only 8Gb total to the backplane. Then the backplane, in turn, has only 8Gb to the other switch. These are just made up numbers, but how would beowulf handle it? Can it group jobs requiring higher communication throughput onto the nodes which are closer to eah other? Does it have to be told the topology, or does it figure it out?
    • Re:Topology? (Score:4, Interesting)

      by tconnors ( 91126 ) on Monday November 25, 2002 @05:50AM (#4750050) Homepage Journal
      What's the bandwidth of that trunk? Also, what's the capaity of the connections between each 16-port card and the backplane?

      Just curious... suppose all the units on a 16-port card have 1Gbps each, but only 8Gb total to the backplane. Then the backplane, in turn, has only 8Gb to the other switch. These are just made up numbers, but how would beowulf handle it? Can it group jobs requiring higher communication throughput onto the nodes which are closer to eah other? Does it have to be told the topology, or does it figure it out?


      Sounds like they have 10 ports @ 1Gbit each free, so get about 10Gbit between the switches. When we [slashdot.org] (position 180 on the latest top500) were investigating thin-tree connections, we thought that we might be able to effectively run one job on a third of the nodes, another on the second third, and miscellaneous jobs on the 3rd (since we have two or 3 researchers who like to use lots of resources, and people like me who only need a single proc at a time). So you just partition the nodes and only allow your mpi jobs to sit on one group at a time.

      Then there are things like openmosix which deal with topology automatically somehow. They will try to calculate the speed of the interconnect and the various nodes' procs (if heterogenous), and work out the distribution in the most efficient way. I am going to try to convince my sysadmin to try out openmosix on the lesser-used nodes of the cluster, because there is a feature of it that I think one of us might like - the combination of memory of the different nodes in one big contigous space, but right now we are busy cleaning up after the upgrade.
  • Hmmm..I'll bet they needed this to figure out how many radioactive trees there are nearby!
  • Looks like they fit 55 shuttles per rack, so 6 racks of space. Very small in terms of size. My Sun 10K's take up 3 racks each.

    Not sure who makes those plain metal racks, but I picked some up at costco(brown box), and they are sweet. They have big caster wheels so you can get to the cables. I use them in the closet, tv rack, and my server rack.

    Setting up the hardware is easy, I'm curious about the clustering software. Wonder if any 3d rendering packages exist (opensource/free) work on a linux cluster.
    -
    "Marijuana? Cocaine? I'm not going to talk about what I did as a child." - President Bush
  • Imagine a raven-cluster of these.. er.. uh.. imagine a rhyme-of-the-ancient-mariner-cluster of these.

    Take all of our fun away by using a beowulf cluster... damned scientists. Everything else just sounds lame.

    The-tyger-cluster? Sounds more like some sorta lame attack. Bah, ferget it.
  • I've followed Mike Warren's earlier Linux clusters with interest: Loki (x86), mid-90's, and Avalon (Alpha) a few years ago.

    The free software and low cost supercomputer are not so much news anymore since every intelligent consumer of compute cycles has at least one of these clusters available. No one has to "imagine" them anymore; they are real and commonplace.

    What's a nice development here is that the Los Alamos team has not only brought down the ratio of

    $/FLOP
    but they've started looking to bring down the ratio of
    Watt/FLOP
    as well.

    It represents an uncharastically appropriate use of resources at the Department of Energy and it also helps point the way for businesses looking to further minimize operational costs of racks of computers in air-conditioned rooms.

  • Wonder what a Beowulf cluster of these things would be like...

    hahaha, I'm so funny. Laugh at my stupid played-out beowulf cluster joke...
  • by Lxy ( 80823 )
    But does it run on radioactive trees? [slashdot.org]
  • Comment removed based on user account deletion
  • "294-unit Beowulf claster..."

    GAHH! Finally a story where the beowolf cluster is an actual part of the story and you misspell it!

If I want your opinion, I'll ask you to fill out the necessary form.

Working...