Pycharm 3.02 on OpenIndiana

The IDE

Working on the Ajax module of Brython (and multilingual documentation)

Requirements

PyCharm, the free community edition and the professional versions simply require a Java JDK. I use the Oracle (Sun) JDK7 on OpenIndiana.

Installation

The only challenge is downloading the tar.gz file. By faking the user agent to firefox for linux, the download link appears:

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0

Then it's a matter of gunzip and tar -xvf and moving the directory to /opt and running pycharm.sh in the bin folder.

Francois
@f_dion

Repair, Upgrade or Replace

[I had decided to not publish this, but with the LX brand revived, I thought it would be an interesting story post]

The perpetual question

Repair, upgrade or replace? It doesn't matter if we are talking about an appliance, a car, furniture or a laptop. Whatever it is, it will break or become obsolete. It's all a question of scale and time, and outside pressures.

What do I mean by outside pressures? Let's say you have a car, it is yours and it is working fine. You drive it 30,000 miles each year. It does 20mpg (US). Let's say gas price is $3.50 a US gallon, that's a tad over $5000 a year to make it run (not counting normal maintenance). A lot of money to go from point A to point B, but still within reason. The outside pressure is minor. But let's bump up gas price to $9.00 a gallon, and we are now talking over $13,000 a year. $27,000 over two years. ROI on getting a new car with twice the mpg is now pretty obvious.

But what if there are no options

In our example above we assume there is such a thing as a 40mpg vehicule that can replace the one you have. But what if I told you it has to be able to tow 6000lbs? There might not be any option that fit the bill. So you have to compromise in one way or another. And that is where it gets complicated.

Lightweight, but not lightweight

If you spend times going from one place to another doing presentations, no doubt you've welcomed tablets and ultra portable laptops. To be able to demo a specific n-tier setup or even a whole datacenter directly on a 13" laptop that weighs less than 3.5 lbs is definitely a great thing.

I've been using for a few years a Sony Vaio Z that is totally custom. Large SSD, large drive, multi core. Ram was the limitation back then, 8GB. But anybody familiar with Solaris or (open source solaris) IllumOS based OSes knows that thanks to cpu and memory sharing between each zones and the full virtual networking capability of crossbow, you can simulate a lot of machines within that 8GB.

My outside pressure

Linux zones are no longer available, so I have to rely on virtualbox or other similar solutions. They dont share ressources such as memory and are not lightweight solutions like zones. 8GB is no longer acceptable. CPU or disk is not a problem (not dealing with big data, just terabyte scale).

Lack of solutions

For some reason there are very few lightweight laptops with > 8GB. 16GB seems to be the next barrier, but typically with a very low screen resolution. Why is that? I don't know. So, for right now I have to choice but to wait on technology to catch up, assuming market demand. [Or, now, hope to get LX branded zones back. If somebody was looking for a good business case, I'm giving you one]



François
@f_dion

LX zones back in illumos and OpenIndiana?

Branded Zones

If you've used Solaris and typed man lx, you saw:

Standards, Environments, and Macros lx(5)NAME

lx - Linux branded zone

DESCRIPTION


The lx brand uses the branded zones framework described in
brands(5) to enable Linux binary applications to run unmodi-
fied on a machine with a Solaris Operating System kernel.

The lx brand includes the tools necessary to install a Cen-
tOS 3.x or Red Hat Enterprise Linux 3.x distribution inside
a non-global zone. The brand supports the execution of 32-
bit Linux applications on x86/x64 machines running the
Solaris system in either 32-bit or 64-bit mode.

Supported Linux Distributions
The lx brand emulates the system call interfaces provided by
the Linux 2.4.21 kernel, as modified by Red Hat in the RHEL
3.x distributions. This kernel provides the system call
interfaces consumed by the glibc version 2.3.2 released by
Red Hat.

In addition, the lx brand partially emulates the Linux /dev
and /proc interfaces.


But if you've followed the story, lx branded zones became a mere historical reference. Until now.

Reintroduction

David Mackay posted the following webrev to the illumos developer mailing list:

http://cr.illumos.org/~webrev/DavidJX8P/lx-zones-restoration/


This is great, and I hope this gets reintegrated and pushed in the next openindiana!

François
@f_dion

The wrong OS

I always cringe when I see a crashed display in an airport, a store etc. That's what happens when you run a PC and Windows for a digital signage solution:

You can see the windows 7 start bar at the left side...

Bluetooth btms_installer.sh

Blast from the past

Not an article today, just a question:

Does anybody know where I can find a copy of btms_installer.sh?

It used to be at:

http://www.opensolaris.org/os/project/bluetooth/btms_installer.sh

François
@f_dion

OI: Python modules that need cc

Tempting

So, it is always tempting to be at the latest greatest of some software. For example, Sun (Solaris) Studio 12 update 3 is freely available to download. But sometimes the latest will leave you with some head scratching.

Python modules

Some Python modules require a C compiler in order to provide a faster runtime, or even to install at all. Numpy is one such module. It provides superb capabilities in Python, but even doing:

# pip install numpy

It will still invoke the C compiler. Due to how the Python interpreter was compiled on OpenIndiana, by using the latest compiler (installed, say, in /opt/solarisstudio12.3), your install will fail. Using the Sun Studio in the opensolaris.org repository (yes, the old 12.1), things work great, as it is installed in /opt/sunstudio12.1

So if you encounter a failed install of a Python module under OpenIndiana, there is a good chance you hit the above problem.

François
@f_dion

dtrace with Python

dtrace: Python instrumentation

...where time becomes a loop

Last year, I mentionned that it was time for the Python community to embrace dtrace. I've gotten questions left and right, at user groups, through email etc as to what is dtrace and how it ties in with Python.

This week, a few posts on the Argentinian and Venezuelan Python lists on debugging Python and a total absence of a mention of dtrace and I knew I had to do a writeup. But before we get into the details, let's step back a bit.

Head over to my Python blog to read the rest of the article.

François
@f_dion

Illumos on RaspberryPi

Starting point

I have now dedicated two Raspberry Pi to work on some bare metal (C, assembler, no operating system) code. I'm hoping that publishing some of this work will help to start somekind of an effort to get the OpenSolaris ARM code out of the mothballs.

IllumOS

Obviously, long term will be to get the IllumOS kernel booting, but there is a lot of work between now and then.

On this blog

So, usually I post all the Raspberry Pi stuff on the Raspberry Pi Python Adventures blog, but I want to keep it mostly Python, with the occasional C code (swig etc). Since we are going bare metal, I think it'll be better if I track this stuff here.


@f_dion

OpenSXCE

Mystery solved

The new open sourced solaris / illumos distribution is called OpenSXCE (as in Open Solaris Express Community Edition - a clin d'oeil to the original open sourcing done by Sun and the Express program)

OpenSXCE

brought to you by Martin Bochnig 30 January 2013
Here is the first release of OpenSXCE for SPARC.
OpenSXCE 2013.01 Live for SPARC (1.2GB) -- 30 January 2013

The site is www.opensxce.org

Note the md5sum : 330a82654405c77c8cea903ed18117f2


@f_dion

Mysterious post

Hint

It has to do with illumos, solaris and sparc

What he said

Martin Bochnig via jeeves.archives.listbox.com
to discuss, developer Hi Garrett,   Release tomorrow. Status: http://twitter.com/MartinBochnig

Till then.

Martin

Git error setting certificate verify location

Problem

Under OpenIndiana, you might have  tried a git clone https://github.com/something....

and got this:

Cloning into something...
error: error setting certificate verify locations:
  CAfile: /etc/curl/curlCA
  CApath: none
 while accessing https://github.com/something....

fatal: HTTP request failed

Solution

The  fix is easy:

su
Password:

mkdir -p /etc/curl && cat /etc/certs/CA/*.pem > /etc/curl/curlCA

LZ4 for ZFS

IllumOS update

Look what just happened last night:

https://www.illumos.org/issues/3035

Description

LZ4 is a new high-speed BSD-licensed compression algorithm written by Yann Collet that delivers very high compression and decompression performance compared to lzjb (>50% faster on compression, >80% faster on decompression and around 3x faster on compression of incompressible data), while giving better compression ratio. For detailed benchmark numbers see the attached spreadsheet.

Updated by Christopher Siden about 16 hours ago

• Status changed from New to Closed

What does it mean?

In plain english, the latest IllumOS kernel has LZ4 compression available for the ZFS filesystem.

Bonjour / avahi / zeroconf

I use a few Raspberry Pi on my network. I gave them proper hostnames. I could map the DNS and all that, but I simply have avahi running on them, so I can access them with hostname.local.

Multicast

For OpenIndiana to support zeronconf, all that is needed is this:

fdion@p490s:~$ su
Password:
fdion@p490s:~# vi /etc/nsswitch.conf

[here, I just added mdns:

hosts:      files mdns dns
ipnodes:    files mdns dns

]
fdion@p490s:~# svcadm enable svc:/network/dns/multicast:default
fdion@p490s:~# exit

From that point on, I can simply use:

fdion@p490s:~$ ssh -X rpi01.local

And of course, I had to configure Avahi on the Raspberry Pi.

This also works with my Macs, I can use ssh mac01.local for example.

Virtual Box with SMF control

SMF

The Service Management Facility (SMF), first introduced in Solaris Nevada, as project Greeline, and then later in the commercial Sun Solaris, is the modern way to manage system and application services in a self healing, fault tolerant way. It replaces the old init mechanism.

Some historical background information (including all the PSARCs) on SMF is still available at opensolaris.org.

Another good read is the Joyent wiki.

VirtualBox

Alexandre Dumont published some scripts for Virtual Box, to integrate with SMF. That was in 2009. Not too long ago, I was trying to find his blog and it was down.

I just stumbled upon:
https://forums.virtualbox.org/viewtopic.php?f=11&t=33249

And there is a sourceforge project for it, here:
http://sourceforge.net/projects/vboxsvc/

Installation

From the README, using the pkgadd in a global zone:

    # gzcat COSvboxsvc-0.16.pkg.gz > /tmp/x
    # pkgadd -d /tmp/x -G

You probably want the -G flag. It doesn't block you from manually installing
the same package in a certain local zone where you'd use VirtualBox, but
it blocks automatic package propagation to those local zones which are
not expected to use and run VirtualBox. For us these zones are rare,
zero or one per machine (there is no definite/hardcoded limit though). YMMV.

To update the package you can simply remove the old version and install
anew, i.e.:
    # gzcat COSvboxsvc-0.16.pkg.gz > /tmp/x
    # pkgrm COSvboxsvc
    # pkgadd -d /tmp/x -G

A cleaner way is to use an admin file to overwrite an existing package,
i.e. one from LiveUpgrade:
    # gzcat COSvboxsvc-0.16.pkg.gz > /tmp/x
    # pkgadd -d /tmp/x -G -a /etc/lu/zones_pkgadd_admin

Also note that this package "depends" on SUNWvbox, so that should be
installed beforehand.

Tribblix

 

 

OpenSolaris

When Solaris was open sourced, several distributions came to life from the OpenSolaris source code. Most of them however relied on closed bits provided by Sun. Then Oracle came and put Han Solo in a carbonite... err, wrong story. But you get the point.

With the emergence of Illumos (opensolaris with 100% open bits), we are starting to see again a surge in number of distributions based on it, and this is great news.

Now, there are several focusing on storage, and of course that is a natural application for an operating system based on Solaris technologies, such as ZFS.

But I want a desktop. I'm currently using OpenIndiana, and it works fine for my needs, better than any other OS on the planet, due to the fact I need to manage largish projects on my desktop. One in particular has over 3/4 million lines of code. Zones (clones) and snapshots are a must to tackle that. And dtrace to debug. But if you've ever used Solaris 10 or above you already know that.

Tribblix

Today, we are looking briefly at Tribblix.

The homepage is at http://www.tribblix.org/

What's in the name? Peter Tribble answers: "

Well, while it may seem rather vain, if I were to choose a name from scratch I would probably have come up with some obscure solar-powered pun. But at one of the OpenSolaris summits we took part in a Go Game, and one of the questions was:

Which of the following is not an OpenSolaris-derived distribution:

• Schillix
• Tribblix
• Belenix
• OpenSolaris 2009.06

I'm not sure that whoever set the question knew how close that was to the name of someone on the project, but that was how the name became planted in my consciousness."

Pretty funny... At that time, there was no Tribblix, but it sparked something, and Peter has now released Tribblix.

Speaking of Sparc

When asked on the illumos discussion list about availability of a Sparc version (Tribblix is x86), Peter indicated the following:

"

Do you have a SPARC ISO for your SVR4 distro, Peter?

Not yet. That shouldn't be too difficult. (It's much easier to develop
and test on x86, as I can repeatedly spin something up in VirtualBox
until I get it right. The differences in building the ISOs for sparc are
relatively small.)

As I recall, though, Martin said he had stripped out the IPS packaging
from his DVD, which I would need in order to convert all the packages."

Peter is referring to Martin Bochnig's own distribution that runs on Sparc systems. I will cover that one in the next blog post.

ISO Please

It is to be found here: http://www.tribblix.org/tribblix-0m1.iso

From OpenIndiana, it is a simple matter of issuing:

$ cdrw -i tribblix-0m1.iso

From any other OS, use your favorite method to burn an ISO. This is a CD image, btw, something rare nowadays.

Install

The installation process is relatively quick, but it is for people who are well versed in the Solaris OS. After choosing your country and language, you'll have to play with format and fdisk before starting the install process.

Instead of going through the steps here I will point you to Peter's install guide.

One thing I'll point out, when you are about to run the install script, make sure you type the following:

/.cdrom/live_install.sh c1t0d0s0 x11 retro-desktop develop xfce

That way, you install x11 etc. Else, you are 100% command line.

Desktop

When you boot Tribblix, you end up with a text login prompt. This is also the approach Raspbian are taking for the Raspberry Pi. I like that.

You login as jack initially (just as you would login as pi on a Raspbian distro). Then you can either startx, which will give you a spartan desktop called TWM. For some that might be enough.

TWM: Spartan and fast

You can also run wmaker.inst to modify your startx to use WindowMaker. I ran this with 512MB of ram, and it runs, a little tight.

WindowMaker

 XFCE

Peter did set it up so you could also start XFCE effortlessly. Instead of typing the typical startx,you would:

$ startxfce4

XFCE4: light yet full featured

What is interesting is that with 512MB of RAM, I still have 108MB free on the above screenshot.

There is potential thus to port this over to the Raspberry Pi. The Pi uses an ARM11 (ARM V6) processor, but OpenSolaris had been ported to ARM in the past: http://hub.opensolaris.org/bin/view/Project+osarm/installation. Another real plus to a port to the Raspberry Pi is that the hardware is pretty much set in stone, so no need for a bazillion sound and video drivers, for example.

Anyway, this concludes our brief review of Tribblix. It is very much a work in progress, but it shows that some interesting stuff can happen still with regards to Solaris and derived distributions.

That is so wrong...

At some point, somehow, somebody thought this would be a good idea...

Lots of new solaris

Oracle Solaris 11.1

First off, we will start with Oracle's new Solaris: Solaris 11.1

So, OK, it is not a brand spanking new OS, but it is an update.

According to Oracle, the new key features are:

• 8x faster database startup and shutdown and online resizing of the database SGA 
• Kernel Mode Acceleration for Oracle Real Application Clusters (Oracle RAC)
• 4x faster Oracle Solaris Zones updates
• Oracle Solaris Zones on Shared Storage

Also, some other new features:

New Logging Daemon

rsyslog, the popular daemon for message logging, has been included in Oracle Solaris 11.1. It is not enabled by default, but administrators can switch to this new logging daemon by disabling svc:/system/system-log:default and enabling svc:/system/system-log:rsyslog using SMF administrative utilities.

Simple SMF Manifest Creation Tool

The svcbundle(1M) command makes it easier to generate SMF manifests and profiles. By providing a few options on the command line, the user can generate a wide variety of manifests and profiles. This relieves the user from the burden of writing XML files.

File System Statistics for Oracle Solaris Zones

Until now, global zone administrators had no way of monitoring activity in a particular zone, nor could non-global zone administrators monitor fstype activity in their own zones. This update provides a per-fstype kstat (kernel statistic) for each zone. The global zone also has a kstat exclusively reporting its activity.

But most of the new features of interest are in the networking stack:

Edge Virtual Bridging (EVB)

EVB extends network virtualization features into the physical network infrastructure allowing users to manage bandwidth and to increase utilization of network resources. EVB is an IEEE standard that defines new protocols and mechanisms for exchanging information about virtual
links between hosts/stations and a switch/bridge. Support for the EVB Virtual Station Interface (VSI) Discovery and Configuration Protocol (VDP) and the Edge Control Protocol (ECP) are new in this release.

Using EVB, properties (such as the bandwidth limit) of the Oracle Solaris Virtual NIC (VNIC) can be exchanged with the switch so that the switch can be configured accordingly for the VNIC.

The standard is defined in the IEEE 802.1Qbg specification. Additional information can be obtained from evb(7P), vdpd(1M), and dladm(1M).

Data Center Bridging (DCB)

DCB allows the same network fabric to be used for both Ethernet and storage traffic reducing overall infrastructure costs in a datacenter. Support for enhanced transmission selection (ETS) based on IEEE 802.1Qaz has been added to the DCB functionality introduced in Oracle Solaris 11 11/11. This will provide guaranteed bandwidth and lossless Ethernet transport for
converged network environments where storage protocols share the same fabric as regular network traffic.

Oracle Solaris Link Aggregation to Span Across Multiple Switches

Link aggregation support in Oracle Solaris 11, being 802.3ad–compliant, does not allow aggregations to span across multiple switches. Thus, an administrator can either work with a single switch configuration (availability might suffer since the switch becomes a single point of failure) or use vendor-specific mechanisms to explicitly configure switches. This feature extends Oracle Solaris link aggregation to provide for a solution that is
independent of the switch vendor to span aggregations across multiple switches.

VNIC Migration

Associations between Physical NICs (PNICs) and Virtual NICs (VNICs) can now be changed without an outage in the network configuration. For example, if a PNIC becomes saturated, a VNIC associated with one key application can be migrated across to another PNIC (one with available bandwidth) without taking an outage.

But, I'm not here to just talk about the now closed source Solaris. There is a lot going on in the open source front too. The next two articles will demonstrate that clearly.

The Sidekick

Wrapping up the week, I just wanted to mention The Sidekick. No, not that phone thing. This one (raspberry-python blog). And the follow up article.

The sidekick is my desktop's sidekick. A secondary computer that complements my workstation. Not in term of speed or anything like that. But in term of adding a whole world of possibilities to interface with all kinds of stuff. Like a vinyl cutter, or a barometer.

Pi. Raspberry Pi

Quite the sidekick this little computer. If you've never heard of it, head out to Raspberrypi.org and read up on it, check the FAQ. Then come back to the Raspberry Pi Python Adventures to learn more.


[edit] forgot to publish this, so here you go.

ZFS send with built in buffer ( mbuffer ?)

Currently attending the live video feed from ZFS day. Somebody made the suggestion to add buffering to zfs send to resolve performance and mismatch issues.

Sounds like a great idea to me.

Over the years, in the past, mbuffer has been suggested as the solution, but not everyone is getting success with that:


[zfs-discuss] zfs send to remote any ideas for a faster way than ssh?

Alasdair offers this recipe with mbuffer that works for some. I'm thinking OpenIndiana specific, because it aborts using some other OSes.


Anyway, zfs send / receive should just work out of the box, so again, a great idea.

NTFS for OpenIndiana

In case you weren't aware, check out NTFS-3G on OpenIndiana. The support forum is here if you have any issues to report. It should just work, though.