No networking after installing Gentoo

After installing a hardened version of Gentoo on Virtualbox, I ran into an issue where the networking just didn’t work. Pings to Google resulted in Network Unreachable.

If you run ifconfig -a, I noticed that eth0 was not there but this weird enp0s3 was. Because of this, I thought that the issue was due to eth0 being missing (some driver issue, etc). Turns out though, the driver was already running and was working. The reason why eth0 doesn’t exist anymore was because of an update in udev which changed the interface names! Now, eth0 is enp0s3!

Anyway, if your networking is still not working in Virtualbox, it turns out that Gentoo somehow didn’t create the net.enp0s3 file, so we’ll just copy the localhost one.

cd /etc/init.d/
ln -s net.lo net.enp0s3

Now restart the network

service net.enp0s3 restart

With that, dhcp connected and I was able to ping google.

Merge Mining with Monocle and Monocle on p2pool with Ubuntu 14.04

The recently released Monocle (MON) Altcoin offers a new opportunity to mine multiple coins at the same time with Monocle.

The new merged mining method is available to people p2pool node operators who install additional software to allow for the coin to be merged mined. For miners, it’s as simple as adding the Monocle address of your choice as your password!

Minimum Requirements

  • Ubuntu 13.04 or greater
  • A server to host your Ubuntu, such as on DigitalOcean
  • Some basic command line interface knowledge

Installing the Monocle Wallet

The first step is to grab the Monocle source or to download the binary from package.

As of today, the Linux package is unavailable for the latest release. Therefore, we will compile from source.

Compiling Monocle from source

First we’ll grab the latest source code from github

git clone https://github.com/erkmos/monocle.git
cd monocle

If you have previously compiled bitcoind or a similar altcoin before, you’ll likely have the prerequisite packages. If not, you’ll need to install these:

sudo apt-get install build-essential libboost-dev libboost-system-dev libboost-filesystem-dev libboost-program-options-dev libboost-thread-dev libssl-dev libdb++-dev libminiupnpc-dev qt5-qmake libqt5gui5 libqt5core5 libqt5dbus5 qttools5-dev-tools

Now you can compile the source

cd src
make -f makefile.unix

Once this is done, you should have monocled in your current working directory!

To maintain consistency with the packaged version of monocled, go ahead and move the binary to /usr/bin/.

sudo mv monocled /usr/bin/

Setting up Monocled as a service

Once you’ve installed monocled, you’ll most likely want to be able to close your terminal without killing monocled itself. The easy option would be to use nohup monocled, but it’s better to run it as a service.

The bitcoin community has a great post on how to set up the bitcoind instance as a service.

To start, create a new file under /etc/init/monocle.conf

description "monocled"

start on filesystem
stop on runlevel [!2345]
oom never
expect daemon
respawn
respawn limit 10 60 # 10 times in 60 seconds

script
user=monocle
home=/home/$user
cmd=/usr/bin/monocled
pidfile=$home/monocled.pid
# Don't change anything below here unless you know what you're doing
[[ -e $pidfile && ! -d "/proc/$(cat $pidfile)" ]] && rm $pidfile
[[ -e $pidfile && "$(cat /proc/$(cat $pidfile)/cmdline)" != $cmd* ]] && rm $pidfile
exec start-stop-daemon --start -c $user --chdir $home --pidfile $pidfile --startas $cmd -b --nicelevel 15 -m
end script

Now go ahead and instantiate the file

sudo initctl reload-configuration

The next step is to create the user that will run monocle. This will help enhance the security of the monocled process by limiting all activity to its own user space.

sudo adduser monocle

Now we will need to create the monocle configuration so that we can start the daemon with the rpc server enabled.
The first step in this is to generate a random server password. You can do that in any number of ways, but the easiest is to run the monocled process.

monocled
Error: To use monocled, you must set a rpcpassword in the configuration file:
/home/monocle/.monocle/monocle.conf
It is recommended you use the following random password:
rpcuser=monoclerpc
rpcpassword=GLGwQQ5RE3KYaBdZgQE4aXdtHapFFiCe2UN9Pnr6tTG9

Go ahead and copy the rpcuser and rpcpassword and paste it into /home/monocle/.monocle/monocle.conf

Now, you should be ready to start up monocled!

sudo start monocled

You can also check on the status of the monocled instance

sudo tail -f ~monocle/.monocle/debug.log

And there you have it! Monocled is now running as a full node on your own Ubuntu server.

Installing the Stratum Proxy software

The next step is to install the startum proxy software to be able to support merge mining.

sudo apt-get install redis-server python-pip mysql-server haskell-platform libmysqlclient-dev python-mysqldb libssl-dev python-redis
sudo pip install bunch python-jsonrpc

We’ll create a new user for added security and create a logging directory for it

sudo adduser proxypool
sudo mkdir /var/log/proxypool
sudo chown proxpool /var/log/proxypool
sudo su - proxypool

Now grab the source code

git clone https://github.com/erkmos/proxypool
cd proxypool

For the proxy pool, you use cabal to install

cabal update
cabal install cabal-install

cabal sandbox init
cabal --force-reinstalls install "aeson >=0.6.2.1" "async >=2.0.1.5" "base16-bytestring >=0.1" "cryptohash >=0.11" "either >=4.1" "hedis >=0.6.3" "hslogger >=1.2.3" "network >=2.4.2.2" "select >=0.4.0.1" "text >=1.1.1.1" "unordered-containers >=0.2.3.3" "vector >=0.10.9.1"
cabal configure
cabal build

Now you’ll have to configure the proxy.

cp proxypool.json.example proxypool.json

Now you’ll need to create the MySQL User to track payouts

mysql -u root -p
CREATE USER 'proxypool'@'localhost' IDENTIFIED BY 'insert-password-here';
create database proxypool;
grant all privileges on proxypool.* to [email protected] identified by 'insert-password-here';
flush privileges;
quit

To configure the payout tracker, open up payout/sharelogger.conf in your proxypool folder

Inside your payout folder, import the sql structure

mysql -u root -p proxypool < db.sql

We’ll also create a service script to run the server as a daemon. Open your favorite text editor to /etc/init.d/proxypool and paste the following contents

#!/bin/sh

DIR=/home/proxypool/proxypool
DAEMON=$DIR/dist/build/server/server
DAEMON_NAME=proxypool
DAEMON_USER=proxypool

# The process ID of the script when it runs is stored here:
PIDFILE=/var/run/$DAEMON_NAME.pid

. /lib/lsb/init-functions

do_start () {
    log_daemon_msg "Starting system $DAEMON_NAME daemon"
    start-stop-daemon --start --background --no-close --pidfile $PIDFILE --make-pidfile --user $DAEMON_USER --chuid $DAEMON_USER --chdir $DIR --startas $DAEMON > /var/log/proxypool/out.log 2>&1
    log_end_msg $?
}
do_stop () {
    log_daemon_msg "Stopping system $DAEMON_NAME daemon"
    start-stop-daemon --stop --pidfile $PIDFILE --retry 10
    log_end_msg $?
}

case "$1" in

    start|stop)
    do_${1}
    ;;

    restart|reload|force-reload)
    do_stop
    do_start
    ;;

    status)
    status_of_proc "$DAEMON_NAME" "$DAEMON" && exit 0 || exit $?
    ;;
    *)
    echo "Usage: /etc/init.d/$DAEMON_NAME {start|stop|restart|status}"
    exit 1
    ;;

esac
exit 0

Mark the script as executable and then run the server

sudo chmod +x /etc/init.d/proxypool
sudo /etc/init.d/proxypool start

You can check the log for debugging if needed

tail -f /var/log/proxypool/out.log

It should say that it is authorized which means it is working.

The last thing to do is execute the payout script in the payout folder

python payout.py

If all is good, you can run it in nohup which will survive your terminal being closed.

nohup python payout.py &

Installing p2pool for Vertcoin mining

p2pool for scrypt-n coins is actually a little different due to the nature of the algorithm differences.

Minimum Requirements

  • A server to host your p2pool instance, such as a server from DigitalOcean
  • Ubuntu 13.04 or greater

Installation

Installing p2pool

First we’ll install the needed packages

sudo apt-get install libpython2.7-dev python-zope.interface python-twisted python-twisted-web

To help secure your p2pool instance a bit more, we’ll create a new user for p2pool and switch over to that user

sudo adduser p2pool sudo
sudo su - p2pool

Grab the latest p2pool source code

git clone https://github.com/donSchoe/p2pool-n.git

Now we’ll compile the vertcoin module

cd py_modules/vertcoin_scrypt
sudo python setup.py install

Running p2pool

Using the rpcuser and rpcpasswd found in your vertcoin.conf, fill in the values in the diamond brackets

./run_p2pool.py <rpcuser> <rpcpassword> --net <network>

You have these options for the Vertcoin mining network:

  • vertcoin1 (hash rate 3mh/s or up)
  • vertcoin2 (hash rate 1.5mh/s or up)
  • vertcoin3 (Hash rate 1.5mh/s or below)

Once you execute the command, your p2pool should be up and running. You can confirm by going to your browser and going to http://<server ip>:9147.

Installing the Vertcoind Full Node on Ubuntu 14.04

I was recently interested in the new ASIC-resistant coin called Vertcoin. For mining purposes, you’ll need a vertcoind instance to run p2pool.

To start, you can grab any Virtual Private Server loaded with Ubuntu 13.04 or greater.

Minimum Requirements

  • Ubuntu 13.04 or greater (This post details how to install using 14.04)
  • A server to host Ubuntu, such as DigitalOcean
  • Some basic knowledge of how to use a Command Line Interface

Downloading Vertcoind

The first step is to either compile Vertcoin from source or to install the packaged vertcoind instance. The latter is a bit easier but the former allows you greater flexibility in what kind of features you can get.

Installing vertcoind from packages

To install Vertcoind from a packge, you’ll need to add a PPA. If you would rather compile from source, skip this step.

sudo add-apt-repository ppa:vertcoin/ppa
sudo apt-get update
sudo apt-get install vertcoind

You’ll now have vertcoind installed and can skip the next step.

Installing vertcoind from source

The first step is to obtain the vertcoin source code. You can obtain the Vertcoin source code from github

git clone https://github.com/vertcoin/vertcoin.git

If you have previously compiled bitcoind or a similar altcoin before, you’ll likely have the prerequisite packages. If not, you’ll need to install these:

sudo apt-get install build-essential libboost-dev libboost-system-dev libboost-filesystem-dev libboost-program-options-dev libboost-thread-dev libssl-dev libdb++-dev libminiupnpc-dev qt5-qmake libqt5gui5 libqt5core5 libqt5dbus5 qttools5-dev-tools

Once you’ve installed the prequisite packages, move to the src directory and compile the vertcoind binary

cd src
make -f makefile.unix

Once this is done, you should have vertcoind in your current working directory!

To maintain consistency with the packaged version of vertcoind, go ahead and move the binary to /usr/bin/.

sudo mv vertcoind /usr/bin/

Setting up Vertcoind as a service

Once you’ve installed vertcoind, you’ll most likely want to be able to close your terminal without killing vertcoind itself. The easy option would be to use nohup vertcoind, but it’s better to run it as a service.

The bitcoin community has a great post on how to set up the bitcoind instance as a service.

To start, create a new file under /etc/init/vertcoin.conf

description "vertcoind"

start on filesystem
stop on runlevel [!2345]
oom never
expect daemon
respawn
respawn limit 10 60 # 10 times in 60 seconds

script
user=vertcoin
home=/home/$user
cmd=/usr/bin/vertcoind
pidfile=$home/vertcoind.pid
# Don't change anything below here unless you know what you're doing
[[ -e $pidfile && ! -d "/proc/$(cat $pidfile)" ]] && rm $pidfile
[[ -e $pidfile && "$(cat /proc/$(cat $pidfile)/cmdline)" != $cmd* ]] && rm $pidfile
exec start-stop-daemon --start -c $user --chdir $home --pidfile $pidfile --startas $cmd -b --nicelevel 15 -m
end script

Now go ahead and instantiate the file

sudo initctl reload-configuration

The next step is to create the user that will run vertcoin. This will help enhance the security of the vertcoind process by limiting all activity to its own user space.

sudo adduser vertcoin

Now we will need to create the vertcoin configuration so that we can start the daemon with the rpc server enabled.
The first step in this is to generate a random server password. You can do that in any number of ways, but the easiest is to run the vertcoind process.

vertcoind
Error: To use vertcoind, you must set a rpcpassword in the configuration file:
/home/vertcoin/.vertcoin/vertcoin.conf
It is recommended you use the following random password:
rpcuser=vertcoinrpc
rpcpassword=GLGwQQ5RE3KYaBdZgQE4aXdtHapFFiCe2UN9Pnr6tTG9

Go ahead and copy the rpcuser and rpcpassword and paste it into /home/vertcoin/.vertcoin/vertcoin.conf

Now, you should be ready to start up vertcoind!

sudo start vertcoind

You can also check on the status of the vertcoind instance

sudo tail -f ~vertcoin/.vertcoin/debug.log

And there you have it! Vertcoind is now running as a full node on your own Ubuntu server.

A date with Java's URL API

We’re currently working on a project that involves signing parameters with HMAC+SHA1. In the Java sense, that is no big deal, you can simply use the Mac API.

After creating the classes and the appropriate unit tests, I went ahead and ran a full test suite, but to my horror I received at least a dozen failed unit tests.

However, when I went back and ran the test suite for the individual classes, the suite passed with flying colors. Now I knew that the issue lied with the interaction between multiple classes and something in the more complicated world of multi-threading and race conditions.

Summary

There is an issue where if Mac.getInstance is called prior to the Smock JaxWsPortClientInterceptor URL creation is made, then you will end up using the live http handler rather than the mocked out one. This will cause it to connect to http://localhost:8080 which will most likely fail.

A little background

The new class that was created served the purpose of signing a set of parameters with the HMAC+SHA1 algorithm.

The failing test suite was a web service class that was being tested using the Smock, which ultimately utilizes spring-ws to do a lot of the heavy lifting.

The stack trace of the failing tests had something along the lines of

org.springframework.remoting.RemoteAccessException : Could not access remote service at [ApiService]; nested exception is javax.xml.ws.WebServiceException: java.net.ConnectException: Connection refused
at org.springframework.remoting.jaxws.JaxWsPortClient Interceptor.doInvoke(JaxWsPortClientInterceptor.ja va:504)
at org.springframework.remoting.jaxws.JaxWsPortClient Interceptor.invoke(JaxWsPortClientInterceptor.java :481)
at org.springframework.aop.framework.ReflectiveMethod Invocation.proceed(ReflectiveMethodInvocation.java :172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

At first glance, the error seems to be related to Spring’s ApplicationContext somehow getting dirty and causing the remote service to connect to an endpoint that does not exist.

Debugging the tests

Where is the bug?

To first determine if the error was due to the injection of faulty code on my part to somewhere in the Spring configurations, I went ahead and ran the test suites independently. The result was a bit infuriating, they both passed. When I ran the two classes sequentially, the error was reproduceable. So it seems that the issue is somewhere in the code I have written.

The first step to finding out where exactly is causing the issue was to try to clean up the Spring context. Spring has a pretty nifty feature if you’re using the SpringJunit4ClassRunner that allows you to reset the application context after it has been dirtied. By using the @DirtiesContext annotation, Spring will clear up the application context either after every method invocation or after the class has completed running.

Unfortunately for my case, even with the @DirtiesContext annotation, the unit tests were still failing with the same stack trace.

Setting up breakpoints

At first thought, the best place to start debugging would be to check where we are attempting to connect to. Not knowing exactly how the spring-ws testing framework is set up, I decided to debug the connection at the JaxWsPortClientIntercepter. When I ran the tests, I found that we were attempting to connect to http://localhost:8080. When I did a curl http://localhost:8080, curl returned to me that the connection was indeed refused.

Now that I found out that was the source of the connection refusal, I checked to see if any process was running that uses port 8080. I ran the test suite standalone (the one that works), and no process ever spawned that listens on 8080. I then reran the failing test combination and still no process ever spawned. So, somehow the framework is actually connecting to the endpoint without having to spawn a process that listens to port 8080!

A little experiment with the Mac class

My attention now turned to the Mac class that was the source of the issue. When I attempted to load the source code into my IDE, Intellij, I was unable to view the source since my JDK wasn’t compiled with debug symbols. After finding a more appropriate JDK, I was able to load the source code. However, nothing was of interest in the method I was calling, Mac.instanceOf(). Therefore, I decided to take a closer look at how JaxWsPortClientInterceptor works.

A closer look at Java’s URL API

In JaxWsPortClientInterceptor, I discovered that the way it connects to an endpoint is by loading up the WSDL and then uses the Java URL API to connect to it.

Thinking that the issue may be in the way URL is being used, I loaded up the source in a debugger. After stepping through the code several times, I found that the way URL works is that whenever it finds a new protocol schema, it will add it to a HashTable of protocol handlers. This protocol handler list is static and thus accesible to all instances of URL.

Knowing this, I loaded up what the HashTable contains on both the failing unit test and the passing ones.

What I found was that in the passing test, the handler for HTTP is ThreadLocalMockHttpUrlConnection whereas when the test fails, it is sun.net.www.protocol.http.Handler. It seems that whenever the failing test combination runs, something is causing the Handler to be set as the Sun implementation of HTTP Handler! This of course causes a live attempt to connect to http://localhost:8080 which will cause the Connection Refused!

Knowing this, I went back to the Mac class and lo and behold, I found where the Handler was being added. Take a look here at JceSecurity. It seems that due to security, they want to statically connect to http://null.sun.com. Due to this, the HTTP Handler gets added and will later on be used to connect to our fake endpoint.

To get around this, there were a couple options.

  1. Somehow clear the handler HashTable
  2. Workaround the whole issue by changing the endpoint in my WSDL from http to https

Since URL didn’t provide any means to clear the table in any clean fashion, I decided that since this is a mere test, I’d simply change the http to https and be done with it. The tests then passed.

So it took about a good day’s worth of work to figure out this out only to realize you can fix it by adding one single character. It’s just one of those days.