Relentless Coding

Kagi Assistant Times Out when on VPN

Sun, 10 Aug 2025 09:36:18 +0200

I use Kagi as my search engine and for LLM interactions. I am also alwaysconnected to ProtonVPN. I and others have noticed that the output from theirLLM Assistant is cut of after a couple of seconds when onProtonVPN. Why does that happen and what can we do about it?

What’s Wrong with VPNs?

Instead of a typical LLM response flow, we see:

At first, the problem seemed to be related to being on a VPN:

vpn$ H=kagi.com; time openssl s_client -quiet -servername $H $H:443 Connecting to 34.111.242.115depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R1verify return:1depth=1 C=US, O=Google Trust Services, CN=WR3verify return:1depth=0 CN=kagi.comverify return:180CB5B021D740000:error:0A000126:SSL routines::unexpected eof whilereading:ssl/record/rec_layer_s3.c:696:real    0m6.599suser    0m0.005ssys 0m0.012s

When using the LLM from a static residential IP, no timeout occurs (I gotimpatient, so terminated with ^C):

residential$ H=kagi.com; time openssl s_client -quiet -servername $H $H:443Connecting to 34.111.242.115depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R1verify return:1depth=1 C=US, O=Google Trust Services, CN=WR3verify return:1depth=0 CN=kagi.comverify return:1^Creal    1m25.933suser    0m0.016ssys 0m0.007s

At first, I thought that maybe ProtonVPN ages out TCP connections after a coupleof seconds of inactivity to free space in its NAT/PAT translation table. Butthat seemed like a draconian expiration, and also, it didn’t bear out:

vpn$ H=nos.nl; time openssl s_client -quiet -servername $H $H:443 Connecting to 18.239.50.102depth=2 C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication RootR46verify return:1depth=1 C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication CA EVR36verify return:1depth=0 serialNumber=32142157, jurisdictionC=NL, businessCategory=PrivateOrganization, C=NL, ST=Noord-Holland, O=Nederlandse Omroep Stichting, CN=nos.nlverify return:1^Creal    0m32.315suser    0m0.007ssys 0m0.009s

Oh, So It’s Google

OpenAI’s o3 model suggested to me that it might be Google’s Load Balancer. Yousee, every ProtonVPN server has a pool of addresses. Some or all of theseaddresses are marked by Google as high-risk, because a lot of simultaneousconnections and/or attacks originate from them to Google properties. To preventcertain kind of attacks (Slowloris), resource depletion), Google tears downconnections when the client is not sending data. Looking at a typical exchangewith Wireshark, you will notice that if the model thinks more than 5 seconds,the connection will be torn down. If the model starts outputting its responsebefore 5 seconds have elapsed, the connection will still be cut down exactly 5seconds after the client sent its last TCP segment with a payload (i.e. theprompt). Although Wireshark shows my client sending many subsequent ACKs, noneof these carry a payload, and they are thus not seen by Google as meaningfultraffic:

I haven’t found any documentation to back up these claims, but the empiricaldata is there.

What Can You Do?

A workaround would be to try another egress IP address by switching to adifferent (Proton)VPN server. Anecdotally, this seems to work, which suggeststhat Google has not put the entire ProtonVPN IP range on its naughty list.

Seeing that Kagi is using Google infrastructure, they might also be able to dosomething about this by increasing this 5s teardown rule for high-risk IPaddresses in their Google console. I do not know whether that is (1) possible or(2) desirable, though, as it might open them up to potential DoS.

Choose Which Cisco OS to Boot

Mon, 26 May 2025 12:25:00 +0200

Let’s have a look at what Cisco OS gets booted at startup and how we caninfluence that deciscion using the configuration register.

The configuration register is a 2-byte field, represented as 4 hex digits, thatyou can set with the config-register global configuration command.

The last nibble of the register is called the boot field. This is the logicit uses:

If boot field = 0, use the ROMMON OS.
If boot field = 1, load the first IOS file found in flash memory.
If boot field = 2-F:
Try each boot system command in the startup-config file, in order, untilone works.
If none of the boot system commands work, load the first IOS file foundin flash memory.
If all other attempts fail, load ROMMON, from which you can perform furthersteps to recover by copying a new IOS image into flash.
– ICND1 Chapter 35

The process visually (image taken from the same source as the quote):

Quality of Service (QoS)

Sat, 24 May 2025 11:14:05 +0200

Let’s have a look at QoS or how to unfairly manage traffic in the face ofcongestion on Cisco devices.¹

What Is QoS and Why Do We Need It?

QoS is concerned with 4 characteristics of network traffic:

bandwidth (capacity of a link)
delay (between sending a packet and the recipient receiving it, or RTT)
variation in delay (jitter)
packet loss

QoS comes into play when the network is not able to process every packet as soonas it received, which is most always. Think of a gigabit data stream coming inon 1 interface and having to leave a 100 Mbps link on the other end (speedmismatch). Or several gigabit links on a switch receiving traffic at capacityall having to use the single gigabit uplink to the ROAS (aggregation point).

QoS is good for when we have periodic congestion. If we are congested all thetime, we simple need more bandwidth.

Some types of traffic (voice, interactive video) are much more sensitive totiming (delay, jitter) and loss than others (web browsing). If we do not haveany QoS, then all the traffic gets the default, best-effort treatment. Thiscomes down to FIFO, where the first packet that arrives is the first packet thatis scheduled to leave the outgoing interface. Heavy downloading can delay oreven drown out voice traffic, resulting in a poor Quality of Experience (QoE)for the end user on a VoIP call.

We have 4 tools to change the QoS characteristics of certain flows in thenetwork:

classification and marking
prioritization through queueing
policing (dropping²) and shaping (delaying through queuing)
congestion avoidance (random drops or ECN)

Classification and Marking

Cisco recommends to classify and mark traffic as close to the source aspossible. Marking traffic on access devices prevents having to classify trafficover and over again on distribution or core devices (which are really meant todo bulk forwarding, not classification). We might also not be willing to trustmarking from end devices, but overwrite any received markings in trusted accessdevices under control by IT (the trust boundary lies in the access switch).

Classification can happen at the datalink layer or the network layer.

Datalink/Layer-2 Marking

Inside an IEEE 802.1q trunking frame, there are 3 bits that can be used to set aClass of Service value: priority bits (also known as Priority Code Point [PCP]).In theory, we could set 8 different priority values. But Cisco says we are notallowed to use the highest 2 values, value 6 and 7, because they are reservedfor network use. So, effectively, we can set any value from 0 through 5, with 0being the default (no priority).

Even if we are not using trunking on an interface, we might still be able to useIEEE 802.1p. That is the same as 802.1q except that the VLAN ID (VID,the last 12 bits of the 4 bytes) is set to all zeroes.

The IEEE has made some broad recommendations on how to set andinterpret these CoS values:

PCP value	Priority	Acronym	Traffic types
1	0 (lowest)	BK	Background
0	1 (default)	BE	Best effort
2	2	EE	Excellent effort
3	3	CA	Critical applications
4	4	VI	Video, < 100 ms latency and jitter
5	5	VO	Voice, < 10 ms latency and jitter
6	6	IC	Internetwork control
7	7 (highest)	NC	Network control

Network-Layer/Layer-3 Marking

The problem with layer-2 CoS is that the markings do not cross routerboundaries. We need something similar in IP and we have it in both IPv4 andIPv6.

The 2nd byte of the IPv4 header is called the Type of Service (ToS) orDifferentiated Services. The 2nd and 3rd nibble of the IPv6 header is called“Traffic class”. The function is the same: the first 6 bits are used to indicatethe Differentiated Services Code Point (DSCP) and the last 2 bits are used forExplicit Congestion Notification (ECN).

Differentiated Services Code Points

Before DSCP was around, the first 3 bits of the ToS field were interpreted to bethe IP Precedence (IPP). These looked a lot like layer-2 CoS. Here, too, 3 bitsmeans 2³ = 8 different priority values and, again, Cisco would notallow us to use 6 and 7, because they are reserved for network use). 6 valuesmight not be enough. IPP is now long deprecated and replaced by DifferentiatedServices (DiffServ) RFCs.

DSCP claims the first 6 bits of the ToS field. 6 bits gives us 2⁶ =64 possible code points.³ But what does a DSCP value mean? Luckily, the IETFgave DSCPs a unique name, a relative priority value and a drop profile to beused by Weighted Random Early Detection (WRED) so that equipment from differentvendors can interoperate and execute appropriate so-called Per-Hop Behaviors(PHBs). PHBs are actions other than storing and forwarding a message, such asdelaying, discarding packets or changing header fields.

There are 21 DSCPs, their name, their value in decimal and in binary:

PHB Name	Decimal	Binary
CS0 (default)	`0`	`000 000`
CS1	`8`	`001 000`
CS2	`16`	`010 000`
…	…	…
CS7	`56`	`111 000`
Expedited Forwarding (EF)	`46`	`101 110`

It’s important to note that the DSCP values from the IETF always have the lastbit set to 0. Looking at the table above, we see that DSCP is backwardscompatible with IPP. For example, the IPP value of decimal 5 (101 in binary)was the highest priority under IPP. Under DiffServ it is called ExpeditedForwarding (EF) and it’s got the same meaning. “CSx” stands for “ClassSelector”.

The table above lists only 9 of the 21 promised DSCPs. The other ones are calledAssured Forwarding (AF) and can take any of these 12 values:

Priority	Low Drop Probability	Medium Drop Probability	High Drop Probability
Class 1	AF11	AF12	AF13
	`001 010`	`001 100`	`001 110`
Class 2	AF21	AF22	AF23
	`010 010`	`010 100`	`010 110`
Class 3	AF31	AF32	AF33
	`011 010`	`011 100`	`011 110`
Class 4	AF41	AF42	AF43
	`100 010`	`100 100`	`100 110`

The priority on the left and the first digit in the AF__ scheme is compatiblewith IPP. The drop probability across the top and the second digit in the AF__scheme indicates how soon traffic labeled with these AF classes is eligible fordropping by the queueing mechanism.

So the second digit gives a queueing algorithm such as Random Early Detection(RED), used for congestion avoidance, the marking it needs. Whenever there aremore than 25 packets queued, for example, and a 26th comes in, any of thepackets marked with AFx3 (second digit 3) have a certain chance of beingdropped.

On IOS, we can classify (match) VoIP and interactive video traffic in class mapswith match statements, for example match protocol rtp audio, match cos <0-7>, match dscp ef and others. For example:

! Create class map for voice traffic (RTP audio)class-map match-any VOICE-TRAFFIC match protocol rtp audio match dscp ef match ip precedence 5 match cos 5! Create class map for video conferencing trafficclass-map match-any VIDEO-CONF-TRAFFIC match protocol rtp video match dscp af41 af42 af43 match ip precedence 4

We mark packets in policy maps:

R1(config)#policy-map DEMOR1(config-pmap)#class VOICE-TRAFFICR1(config-pmap-c)#set ?  atm-clp        Set ATM CLP bit to 1  cos            Set IEEE 802.1Q/ISL class of service/user priority  cos-inner      Set Inner CoS  discard-class  Discard behavior identifier  dscp           Set DSCP in IP(v4) and IPv6 packets  fr-de          Set FR DE bit to 1  fr-fecn-becn   SET FR FECN-BECN  ip             Set IP specific values  mpls           Set MPLS specific values  precedence     Set precedence in IP(v4) and IPv6 packets  qos-group      Set QoS Group  vlan-inner     Set Inner Vlan

Here, we can set cos <0-7>, set dscp <0-63> or with an IETF mnemonic, orset precedence <0-7> or with a mnemonic.

Congestion Avoidance with Explicit Congestion Notification (ECN)

Now, instead of forcing a TCP Slow Start by dropping packets, it might be betterto just ask a sender to slow down. That’s what ECN is all about.

After enabling ECN on Linux (see below), notice that in the initial TCP SYNsegment, the sender will not set the ECN-Capable Transport (ECT) bit of 10⁴in the IP header. Instead, it will set the Congestion Window Reduced (CWR) andExplicit Congestion Echo (ECE) flags in the TCP header. If the server supportsECN, it will send back a TCP SYN+ACK segment with the ECE flag set. Finally,when the client sends its next segment, completing the TCP 3-way handshake, nowit will set the ECT bit in the IP header (10).

Then, when a router is congested or about to be congested, it can set theCongestion Experienced (CE) bit to 1 and send the package onwards to itsdestination. The recipient will then set the ECE TCP flag in its ACK back to thesender. The sender notices the ECE bit, reduces its congestion window by halfand sets the CWR flag in its next segment, notifying the receiver that thetraffic flow will slow down a bit.

On Linux, temporarily enable⁵ with:

# sysctl net.ipv4.tcp_ecn=1

To see it in action, for IPv4, filter with:

# tcpdump 'ip[1] & 0x3 != 0'

For IPv6, filter with:

# tcpdump 'ip6[1] & 0x30 != 0'

Filtering the TCP flags under IPv4 in tcpdump is straightforward:

# tcpdump 'tcp[13] & 0xc0 != 0'

However, this does not show anything when the network protocol is IPv6,because IPv6 can use extension headers. Those extension headers are indicated inthe IPv6 Next Header field, creating a chain of headers that tcpdump iscurrently (version 4.99.5 from April 8 2025) unable to follow. If you know thereare no extension headers involved, ip6[53] & 0xc0 != 0 will address the TCPflags byte. (40 bytes for the IPv6 header. Unreserved flags are the 14th byte inthe TCP header. And since we are counting from 0: 40 + 14 - 1 = 53.)

We can configure a Cisco router to set the CE bit by configuring WRED withrandom-detect ecn from policy-map class config mode (see config examplelater). I’m not sure, but it may very well be the case that traffic marked withECN, instead of being dropped, always gets the CE bit set when the minimum queuedepth threshold is exceeded. Other, non-ECN traffic is still subject to randomdrops.

Prioritizing Traffic using Queuing

Whenever a router receives traffic faster than it can send it out of a egressqueue, the packet gets stored in this queue and has to wait for bandwidth tobecome available. A queue is a buffer and has finite capacity. We can havemultiple queues based on our traffic classes. A queuing algorithm determineswhat packet to send next from one of the queues.

An old queuing algorithm supported by Cisco is Weighted Fair Queuing (WFQ)where no single flow of traffic can dominate the bandwidth. Notice that thisworks per-flow, not per-host. A single host with lots of connections can stilldominate the bandwidth and starve other hosts that have less flows. WFQ is stillrecommended for the class-default⁶: all traffic that is not otherwiseclassified.

Class-Based Weighted Fair Queuing (CBWFQ) guarantees minimum bandwidth forup to 11 (maximum recommended) classes of traffic. During times of congestion,that minimum bandwidth is guaranteed. If there is no congestion, a class oftraffic could use more bandwidth if it needs it. So we could have a queuecontaining bulk data and guarantee that class 10 or more Mbps.

I’m not completely sure, but according to Cisco documentation,CBWFQ is configured on an interface automatically when a service policy isapplied to the interface. See configuration example later on.

A normal queue is created by the bandwidth <kbps> keyword or bandwidth percent <n>or bandwidth remaining percent <n> in policy-map class config mode.

The queuing algorithm can also handle a single priority queue. This is usefulfor latency-sensitive applications such as VoIP or video conferencing. Such aLow-Latency Queue (LLQ) gets priority treatment up to a maximum bandwidthduring times of congestion. The maximum prevents the LLQ from starving all otherqueues when the egress is congested. Excess traffic is dropped by a policer.

A priority queue is created by using the priority <kbps> keyword (or the[remaining] percent <n> alternatives) in policy-map class config mode.

Congestion Avoidance / Drop Management

Queuing is congestion management (what to do in the face of congestion?).Congestion avoidance are strategies to avoid filling queues to capacity to beginwith. When a queue is full, it start tail-dropping packets: every new packetthat comes in will be dropped. For a protocol like TCP, this will decrease thecongestion window: the sender will reduce the amount of traffic it sends beforewaiting for an acknowledgement (AKA TCP Slow Start). If all TCP flows experienceTCP Slow Start at the same time and trigger their congestion control strategybecause of tail drop, it is called Global Synchronization. This is aninefficient use of bandwidth.

This is where a congestion-avoidance mechanism such as Weighted Random EarlyDetection (WRED) comes into play. It starts randomly dropping some packets whenthe queue depth (amount of packets in a queue) exceeds some defined threshold.

0                 25                 50                 75|---transmitted---|---some dropped---|xxx all dropped xxx

Let’s say the minimum threshold is 25 packets. Any packets exceeding that number(26th and on), will be subject to random drops. The drops become increasinglylikely the deeper the queue becomes. The maximum threshold is 50. Any morepackets are not queued, there are unceremoniously dropped.

Randomly dropping some packets triggers the congestion response from some TCPflows (and presumably QUIC flows), so that those flows slow down, whilepreventing Global Synchronization.

Enable WRED with random-detect in policy-map class configuration mode (seeconfig example below).

Policing and Shaping

Both policing and shaping have the same goal: to prevent traffic from being senttoo rapidly.

Queuing sets a minimum amount of bandwidth we can use; policing and shaping setmaximums. Policing drops traffic that exceeds the speed limit we set²; shapingdelays traffic by temporarily putting it in a queue.

Both policing and shaping use a “token bucket”. The following formula governsthe token bucket:

CIR = B_c / T_c

CIR stands for Committed Information Rate and is the average speed overthe interval of a second (the total amount of traffic sent per second).

B_c (Committed Burst) is the number of bits deposited in the tokenbucket per second divided by the T_c timing interval. If we wantto send 64 kbps and the timing interval is 8, the bucket replenishes with 64,000/ 8 = 8,000 bits per timing interval (or 1/8 of a second). The egress stillsends at line speed (it has no choice), so it will send those bits out as fastas it can and just be silent the rest of the time.

Configuring QoS on a Cisco Router

Cisco uses something called a Modular QoS CLI (MQC). It consists of 3 steps:

Create class maps that are used to match the traffic we want to apply QoS to.
Create one or more policy maps that take the previously defined class maps toapply a mechanism to such as shaping or policing.
Finally we apply the policy map, usually to a physical interface (but couldbe an SVI, for example).

Create Class Maps that Match Traffic

R1(config)#class-map match-any EMAILR1(config-cmap)#match protocol smtpR1(config-cmap)#match protocol pop3R1(config-cmap)#match protocol imap

Here, we create a class map with a made-up name EMAIL (this can be anystring; I make it uppercase so that it stands out from any Cisco keywords). Bydefault, all the selectors (match statements) must match (logical AND):change it to logical OR by setting match-any.

Then, in class map config mode, we can use something called Network-BasedApplication Recognition (NBAR) to match the traffic we want grouped in thismap. There are a lot of matching options:

R1(config-cmap)#match ?  access-group         Access group  any                  Any packets  application          Application to match  class-map            Class map  cos                  IEEE 802.1Q/ISL class of service/user priority values  destination-address  Destination address  discard-class        Discard behavior identifier  dscp                 Match DSCP in IPv4 and IPv6 packets  fr-de                Match on Frame-relay DE bit  fr-dlci              Match on fr-dlci  input-interface      Select an input interface to match  ip                   IP specific values  metadata             Metadata to match  mpls                 Multi Protocol Label Switching specific values  not                  Negate this match result  packet               Layer 3 Packet length  precedence           Match Precedence in IPv4 and IPv6 packets  protocol             Protocol  qos-group            Qos-group  service              Service Instance to match  source-address       Source address  vlan                 VLANs to match

protocol matching might simply look at the destination port of the traffic todecide whether or not it should be in this class map. Ubiquitous TLS has madeDPI pretty much a no-go these days, I would imagine. Although, match protocol netflix is available, so who knows?

Create Policy Maps that Use Class Maps to Police or Shape Traffic

R1(config)#policy-map DEMOR1(config-pmap)#class EMAILR1(config-pmap-c)#police 256000

Here, we create a policy map with the made-up name DEMO. Again, this can beany string; I simply made up DEMO and uppercased it to avoid confusion withCisco keywords.

Then, we select the class map to apply the policy function to, in our caseEMAIL. We police SMTP, POP3 and IMAP traffic to 256 kbps in times ofcongestion. It’s probably a better idea to apply shaping (delaying) and notpolicing (dropping) to email traffic, but this is just for the sake of a simpleexample.

Here, we could also set WRED with the random-detect keyword and ECN withrandom-detect ecn.

Apply Policy Maps to Interfaces

Finally, we apply the policy map to the outgoing side of an interface:

R1(config)#int gig 0/2R1(config-int)#service-policy output DEMO

All email traffic that leaves interface GigabitEthernet0/2 is now policed to amaximum of 256 kbps in time of congestion.

Note that the CBWFQ queuing algorithm is automatically applied.

Verify Our Work

Show all defined class maps:

R1#show class-map Class Map match-any class-default (id 0)   Match any  Class Map match-any EMAIL (id 1)   Match protocol smtp   Match protocol pop3   Match protocol imap

Show all defined policy maps:

R1#show policy-map  Policy Map DEMO    Class EMAIL     police cir 256000 bc 8000       conform-action transmit        exceed-action drop

Show statistics about how are policy map is applied on an interface:

R1#show policy-map interface gig 0/2 GigabitEthernet0/2  Service-policy output: DEMO    Class-map: EMAIL (match-any)        0 packets, 0 bytes      5 minute offered rate 0000 bps, drop rate 0000 bps      Match: protocol smtp        0 packets, 0 bytes        5 minute rate 0 bps      Match: protocol pop3        0 packets, 0 bytes        5 minute rate 0 bps      Match: protocol imap        0 packets, 0 bytes        5 minute rate 0 bps      police:          cir 256000 bps, bc 8000 bytes        conformed 0 packets, 0 bytes; actions:          transmit         exceeded 0 packets, 0 bytes; actions:          drop         conformed 0000 bps, exceeded 0000 bps              Class-map: class-default (match-any)        0 packets, 0 bytes      5 minute offered rate 0000 bps, drop rate 0000 bps      Match: any

Books to Read

This post just barely touched the surface of configuring QoS on Cisco devices.The following books came recommended if you really want to get started:

Enterprise QoS Solution Reference Network Design Guide
End-to-End QoS Network Design, 2nd edition (Cisco, 2013)

This post was inspired by Kevin Wallace’s CCNA videos about QoS andthe official CCNA 200-301 study guide by Wendell Odom. The pictures areblatantly stolen from Kevin Wallace’s video and the official study guide; Ihope not enough people read this blog for that to matter. Otherwise I haveto create my own pictures and I am pressed for time currently because myCCNA exam is soon. You could either buy the videos from Kevin Wallacedirectly or subscribe to O’Reilly. Highly recommended. ↩︎
Or it may just (re)mark the traffic or do nothing and just measure it forlater reporting. ↩︎ ↩︎
Cisco has a design recommendation to not use over 11 different classes oftraffic. ↩︎
In reality, the ECN bit pattern of 10 and 01 are treated the same.Both indicate that the sender is capable of doing ECN Transport. To becomplete, 00 means that the sender is not capable, and 11 means that thesender is capable and a router was congested and set the other bit to 1 aswell. ↩︎
On Arch, make permanent by adding to /etc/sysctl.d/99-sysctl.confand load it with sysctl --system. ↩︎
Create a policy-map <WORD> and then class class-default and thenfair-queue to enable WFQ for all unclassified traffic. ↩︎

OSPF

Fri, 16 May 2025 12:58:33 +0200

What follows is basically a summary of what I have learned about OSPF during myCCNA studies.

Routing Protocols and OSPF

OSPF is a routing protocol. Version 2 is only concerned with IPv4, whereasversion 3 added IPv6 support.

OSPF is an Interior Gateway Protocol (IGP).¹ Its task as a routing protocolis to make sure all routers in an autonomous system (AS; network under singleadministrative control) can route packets to all networks inside that AS andpossibly to the internet. This is contrasted to an Exterior Gateway Protocol(EGP) such as BGP (Border Gateway Protocol) that is concerned with advertisingroutes between ASs.

OSPF is a link-state protocol. It calculates the cost of an outgoing link²and uses that information to determine the best route from source todestination. This is opposed a distance-vector protocol such as RIP. (“Distancevector” is fancy-talk for counting the number of routers [hops] between a sourceand a destination network.) RIP does not take into account the throughput oflinks; if you have a 1-hop connection to the destination subnet over a very slowlink or a 2-hop connection over gigabit links, the RIP process will put the veryslow link in forwarding table (FIB)³.

Enabling OSPF

When you start an OSPF process on a router with router ospf <pid> in globalconfiguration mode, it will dynamically discover any OSPF neighbors. It doesthis by sending out OSPF Hello packets. If the settings of the first Hellomatches the OSPF settings in the neighboring router that is also running OSPF,it will send back an OSPF Hello. After this handshake, a 2-way state isreached.⁴

Each router has a router ID (RID). The RID is either

explicitly configured in router OSPF submode with router-id <rid>; or
the router selects the highest/greatest IPv4 address on an up/up loopbackinterface (int loopback <n> from global configuration mode and then ip address <address> <mask>); or
the highest/greatest IPv4 address on another up/up non-loopback interface.

If the RID is not configured explicitly, and there aren’t any interfaces, youcannot start an OSPF process.

Exchanging Routing Information Databases

Once the router has a neighbor relationship with another router, they will startexchanging routing information. Each router contains a link-state database(LSDB) containing link-state advertisements (LSAs) of several types, 3 of whichwe are currently concerned with for the CCNA exam:

Type 1 LSAs contain information about neighboring routers and the cost of theoutgoing interface to reach those neighbors and the type of connection(broadcast, point-to-point).
Type 2 LSAs are only created on broadcast links by the DR and contain theDR’s IP address and a list of all the other routers on the segment.
Type 3 LSAs are generated by Area Border Routers (ABRs) that are part of 2 ormore areas and summarize networks in another area. (ABRs are discussed lateron in this post.)

On a point-to-point link (having only 2 routers ever on the same link), a masterand a slave are chosen. The master initiates the exchange of databaseinformation by sending a database description (DBD) message to the slave. TheDBD contains only header information, not the actual data structures containingall information. The slave looks at the DBD and, if it finds it misses someroute information, sends link-state requests (LSRs) to the master. The masterresponds with link-state update packets containing the actual LSAs. The slavealso sends the headers of its database to the master, and the same processrepeats in the other direction.

During this exchange, the routers will transition from the 2-way state toExStart, Exchange, Loading and finally the Full state.

Network Types

In the most simple case, on a serial link, there is just a point-to-point linkbetween 2 routers. The routers become neighbors and exchange BDBs.

On Ethernet links, however, by default the broadcast network type is used.⁵Because more than 2 routers can connect to the same subnet, the number ofdatabase exchanges grows exponentially if each router has to exchange itsdatabase information with every other router. This mesh will have n * (n-1) / 2 connections. To combat this exploding overhead, on a broadcast medium such asEthernet, a Designated Router (DR) and a Backup DR (BDR) are elected⁶. Eachrouter still forms neighbor relationships with every other router, but will onlyexchange database information with the DR and BDR. This reduces the number ofexchanges to 2n - 3. If you have 48 routers segment, a full mesh requires48 * (48-1) / 2 = 1128 exchanges, whereas with a DR and BDR you would needonly 2 * 48 - 3 = 93 exchanged. Huge savings.

What happens when you set one interface to a point-to-point network and leavethe other to broadcast? Confusingly, they will still form a full adjacency andexchange their LSDB, but the routes will not show up under show ip route ospf.⁷

Also, if an interface is known to never encounter another OSPF-enabled router onits link, you can set the interface to passive mode with passive-interface <type> <number> in router OSPF submode. It will still advertise this subnet toother routers over other interfaces, but it will not bother to send Hellos out.

Configure OSPF

Enable OSPF on the router:

R1#router ospf 1

The PID is locally significant: it does not need to be the same on anotherrouter to form a neighbor relationship.

There are 2 ways to configure OSPF on an interface: an interface subcommand anda router OSPF subcommand. The first method more intuitive:

R1(config)#interface gig 0/0/1R1(config-if)#ip ospf 1 area 0

The 1 refers to the locally-significant process ID of OSPF. It could beanything between 1-65535 as long as it matches the router ospf <pid>. I haveyet to understand why you would want to have multiple OSPF processes running,but know that you can.

area 0 indicates which area this interface belongs to. Multi-area OSPF becomesinteresting as a way to reduce the time needed to run Dijkstra’sAlgorithm. An (old) recommendation is to start using multiple areasonce you have more than 50 routers in a single area. If you have multiple areas,then one area needs to be area 0 and all other areas need to connect to it.Routers on the border of areas are called Area Border Routers (ABRs). They sendroute aggregation (Type 3) messages advertising that all traffic to specificsubnets need to go through them. Instead of learning the topology of those otherareas, routers just need to learn those aggregated routes.

The old way to specify which interfaces should be included in OSPF is thenetwork <network> <wildcard-mask> area <area-number> router OSPF subcommand.For example, if we have interface g0/1 and g0/2 with IPv4 addresses 192.1.0.1/24and 192.2.0.1/24, respectively, we would include these interfaces with thefollowing network commands:

R1(config)#router ospf 1R1(config-router)#network 192.1.0.0 0.0.0.255 area 0R1(config-router)#network 192.2.0.0 0.0.0.255 area 0

Crucially, the network command does not advertise these routes. It just says:“if there are any interfaces that match this network and wildcard mask,advertise their networks”. This becomes more clear if we use a wildcard mask ofall zeroes:

R1(config-router)#network 192.2.0.1 0.0.0.0 area 0

This would only enable OSPF on the interface that matches the given IPv4 addressexactly. (Conversely, to include all interfaces, use network 0.0.0.0 255.255.255.255 area <n>.) Again, the router will not just advertise192.2.0.1/32 in this case, but whatever the subnet happens to be configured onthe matching interface (for example 192.2/16).

Related this this, an interface needs to have OSPF enabled before its subnet isadvertised over other links. If we forget to enable OSPF on a link, the subnetis not included. Other routers in the area will not know how to send trafficthat way. So, if we have a subnet with no other routers on it and thereforetrying to form neighbor relationships does not make sense, we still need toenable OSPF on the interface. To prevent the overhead of OSPF Hellos, we canconfigure the interface to be “passive” with passive-interface <type> <number>in router OSPF mode.

By the way, a wildcard mask (also used with ACLs), is the inverse of a subnetmask. To quickly calculate a wildcard mask, take the dotted-quad broadcastaddress of 255.255.255.255 and subtract the subnet mask. In our case, thesubnet mask is 255.255.255.0:

  255.255.255.255- 255.255.255.0-----------------    0.  0.  0.255

A previous post has more details on how to interpret wildcardmasks.

Verify

Show the OSPF neighbors:

R1#show ip ospf neighborNeighbor ID     Pri   State           Dead Time   Address         Interface172.16.12.2       1   FULL/BDR        00:00:37    172.16.12.2     GigabitEthernet0/0/0172.16.13.3       1   FULL/BDR        00:00:37    172.16.13.3     GigabitEthernet0/1/0172.16.14.4       1   FULL/DR         00:00:37    172.16.14.4     GigabitEthernet0/2/0

This shows the RID, the priority, the state and function of the OSPF neighbors,respectively, as well as the interface off of which the neighbor lives. Here,the neighbors use the numerically highest IPv4 address as their RID. A state ofDR or BDR means the neighbor has that role, not the current router. Forexample, the neighbor with RID 172.16.12.2 is a BDR.

To see which interfaces on the router have OSPF enable and what role thoseinterfaces have:

R1#show ip ospf interface briefInterface     PID   Area      IP Address/Mask          Cost  State  Nbrs F/CGig0/2/0        1   0       172.16.14.1/255.255.255.0   1      BDR  1/1Gig0/1/0        1   0       172.16.13.1/255.255.255.0   1       DR  1/1Gig0/0/0        1   0       172.16.12.1/255.255.255.0   1       DR  1/1Gig0/0          1   0        172.16.1.1/255.255.255.0   1       DR  0/0

R1 is the DR on Gig0/1/0, for example.

Show router learned through OSPF:

R1#show ip route ospf     172.16.0.0/16 is variably subnetted, 10 subnets, 2 masksO       172.16.2.0 [110/2] via 172.16.12.2, 00:06:57, GigabitEthernet0/0/0O       172.16.4.0 [110/5] via 172.16.12.2, 00:06:57, GigabitEthernet0/0/0

An administrative distance (AD) of 110 is the default believability used byCisco gear for routes learned through OSPF (some other ADs are 0 for directlyconnected routes, 1 for statically configured routes, 90 for EIGRP). The valueafter the forward slash / indicates the routing-protocol specific metric. Thisis the cost OSPF calculated to take us to this subnet.

Troubleshooting

Routers do not become neighbors

The routers are in different areas. An Area Border Router is the border,meaning it is in 2 or more areas, but still needs to be in the same area asanother router to form neighbor relationships. See area settings with show running-config | section ospf, show ip ospf interface [brief] or show ip protocols.
The RIDs are not unique; 2 or more routers have the same RID.

No OSPF Routes Show Up in the FIB

The MTU might be configured differently. List the MTU with show interfaces <type> <number>.
One router has network type broadcast, whereas the other has point-to-pointconfigured. show ip ospf neighbor lists the state of the neighbor.

“Gateway” is another word for “router”. ↩︎
Influence that cost with the ip ospf cost <1-65535> or the bandwidth <1-10000000> interface subcommand, or the auto-cost reference-bandwidth <bandwidth-in-mbps> command in router OSPF mode. The last method ispreferred over setting the bandwidth on individual interfaces. Thereference bandwidth is used in the formula ref_bandwidth / int_bandwidth.Setting it to 100,000 Mbps will result in a cost of 100 for Gigabit Ethernetinterfaces. On modern hardware, this makes sense. It is not recommended tochange the bandwidth setting on individual interfaces. Although this hasno effect on the actual speed of the interface, it might be used by morethan just OSPF. ↩︎
A router contains both a routing information base (RIB) and a forwardinginformation base (FIB). The RIB contains all the routes learned by therouter. These could be directly connected routes, statically configuredroutes and dynamically learned routes. The router then selects the bestroutes to a destination, ignoring worse, duplicate routes. ↩︎
2 routers become neighbors if:
1. Their interfaces are up/up
2. ACLs do not filter routing protocol messages
3. Interfaces are in the same subnet
4. They can authenticate to each other
5. Hello and hold/dead timers match
6. RIDs are unique
7. Interfaces are in the same area
8. OSPF process is running (not shutdown)
9. Interfaces have the same MTU
10. Interfaces have same OSPF network type (broadcast, point-to-point)
↩︎
Change with ip ospf network {broadcast | point-to-point} in interfacemode. ↩︎
Influence the election by changing the RID to a numerically higher numberor setting the priority of the router with ip ospf priority <0-255> ininterface mode. That number defaults to 1. A higher number means higherpriority (contrast that with STP where a lower number means higherpriority). Setting the value to 0 means “do not participate in election”. ↩︎
There are more network types (non-broadcast multi-access [NBMA],point-to-multipoint, point-to-multipoint non-broadcast), but that is outsidethe scope of the CCNA exam, so we are not interested at this point. ↩︎

Convert Cisco Lightweight AP to Autonomous

Sun, 04 May 2025 07:09:27 +0200

If you have a Cisco AP that has CAP in the model, you have a so-calledlightweight AP that is supposed to be controlled by a wireless controller (WLC).I was able to convert my AIR-CAP2602E-E-K9 to autonomous mode by loadingdifferent firmware. This post discusses how to do that.

A show version indicated that I had ap3g2-k9w8-* running – a CAP image. Ineeded to download a ap3g-k9w7-* (notice the 7 instead of 8) image and getit on the AP.

Instructions:

First, start a TFTP server on the same LAN the AP lives on and serve the desiredfirmware image. Then, remove power from lightweight AP. Now, press and hold themode button. Finally, apply power and wait 25 seconds until the status light onthe front turns to solid red. If you only see green, you did not push themode button correctly. (If you just want to reset the configuration to factorydefaults, wait until you are in recovery mode and type, delete flash:config.txt.bak and then reset.)

The console confirms how long you pressed the button:

button is pressed, wait for button to be released...button pressed for 24 seconds

The AP will now try to download tftp://255.255.255.255/ap3g2-k9w7-tar.default.I was, however, unable to see that IP broadcast in tcpdump on my Linuxmachine, and so the connection attempt timed out. I was then shuttled intorecovery mode. Here, I first made sure the things I need were loaded, such as anEthernet connection, the flash file system and TFTP.

ap: ether_initap: flash_initap: tftp_init

Then, I set my IP address and subnet mask. If you connect to the AP via aswitch, they need to be on the same VLAN/network as the TFTP server. My TFTPserver answered to 10.0.0.2.

ap: set IP_ADDR 10.0.0.1ap: set SUBMASK 255.255.255.0

Now, we can download and extract the image in one go:

ap: tar -xtract tftp://10.0.0.2/ap3g2-k9w7-tar.153-3.JF10.tar flash:

If there is not enough space, format flash: first, after you have made sureyou have a backup of the original firmware image.

Finally, let’s set the boot image and reboot:

ap: set boot flash:ap3g2-k9w7-mx.153-3.JF10/ap3g2-k9w7-mx.153-3.JF10ap: boot

Cisco LAN Routing

Thu, 24 Apr 2025 11:23:15 +0200

Let’s look at several ways to route between VLANs in the Cisco world.

Router on a Stick (ROAS)

A router is connected to a switch. If separate VLANs on separate IP subnets wantto talk to each other, they have to go through the router. The router has a VLANtrunk to the switch. The router itself needs an IP address in every VLAN itroutes. These IP addresses are the default routes for the end hosts.

On the router, you create subinterfaces by putting a .<n> after the interfacetype and number, where n can be any number. By doing so, you immediatelycreate the interface.

You tell the subinterface which VLAN ID all incoming frames are tagged with andall outgoing frames will be tagged with:

L3(config-if)#encapsulation dot1q 20

You then assign an IP address to that interface:

L3(config-if)#ip addr 10.10.20.1 255.255.255.0

Finally, make sure the physical interface is not shutdown.

L3(config)#int g0/0/0L3(config-if)#no shutdown

Verify:

Router#sh ip int brInterface               IP-Address      OK? Method Status                Protocol GigabitEthernet0/0/0    unassigned      YES unset  up                    up GigabitEthernet0/0/0.10 10.10.10.1      YES manual up                    up GigabitEthernet0/0/0.20 10.10.20.1      YES manual up                    up GigabitEthernet0/0/1    unassigned      YES unset  administratively down down Vlan1                   unassigned      YES unset  administratively down down

Switched Virtual Interfaces (SVIs)

Layer-3 switches do both layer-2 switching and layer-3 IP routing. When a framecomes in, it will act like a switch, unless the MAC address on the frame isaddressed to the switch itself. In that case, the switch will de-encapsulate theframe and inspect the package. If addressed to the switch itself, it will passthe package to a locally running process, else it will perform its routingfunction and forward the package out one of its interfaces.

The created SVI is not attached to any physical port. Instead, whenever a frametagged with the right VLAN ID comes in on any of the ports, the switch willtreat it as if it came in from that virtual interface. It will switch the frameto any other interface associated with that VLAN ID. The creation of a virtualinterface will give the layer-3 switch’s routing logic an entrypoint into theVLAN.

In order for a layer-3 switch to perform layer-3 routing:

L3(config)#ip routing

Configure the VLAN interface and assign an IP address to it:

L3(config)#vlan 10L3(config)#int vlan 10L3(config-if)#ip address 10.10.10.1 255.255.255.0

Verify:

L3#sh ip route | i 10.10.10C       10.10.10.0/24 is directly connected, Vlan10

If the routing table is empty, you might have forgotten to enable ip routing.

Other gotchas:

VLANs must be defined on the local switch.
The switch must have at least 1 up/up interfaces using the VLAN (up/up accessinterface using that VLAN or a trunk interface for which the VLAN is allowedand is STP forwarding).
vlan <n> must be no shutdown.
int vlan <n> must be no shutdown.

Layer-3 Switch Routed Ports

If we want to route traffic between 2 devices that do routing, we might not needVLAN information. In that case, we can use routed interfaces.

A layer-3 switch can disable the switching behavior of a port (no switchport).This makes it a routed port. It will no longer be forwarding frames based ontheir MAC address. Instead, it will only accept frames addressed to the routedport, de-encapsulate the frame and take it from there.

Configure:

L3(config)#int g1/1/1L3(config-if)#no switchportL3(config-if)#ip addr 10.100.100.1 255.255.255.252

You can verify in multiple ways:

L3#sh int statusPort      Name  Status       Vlan       Duplex  Speed Type! ... snip ...Gig1/1/1        connected    routed     auto    auto  10/100BaseTX

show interfaces should show an IP address (because switch ports do not show IPaddresses on a physical interface).

show ip route should show the physical interface as an outgoing interface inroutes (again, switch ports are not listed as outgoing ports).

show interfaces <type> <number> switchport should show short output confirmingthe port is not a switch port.

Interface Gig1/1/1 has routed under Vlan, indicating that it does notparticipate in any VLANs.

(Make a router port a switch port again by issuing the switchport command ininterface configuration mode.)

Incidentally, you can create a layer-3 EtherChannel bycreating a channel-group <n> mode <active|desirable|on> on 2 or more routedports. Make sure to also issue no switchport on the int port-channel <n>itself and double check that both speed and duplex are set to auto on thephysical ports.

EtherChannel

Thu, 24 Apr 2025 08:55:30 +0200

Let’s have a look at how we can use redundant layer-2 links between 2 Ciscodevices.

Benefits of Using EtherChannel

A layer-2 EtherChannel aggregates up to 8 links. This provides redundancy,because 1 of the links can fail or go down, and traffic will still flow over theother links. It also provides greater bandwidth, which is especially importantover a trunk link, such as in a Router-on-a-Stick (ROAS) configuration.

Without an EtherChannel, the redundant links would still work and, in case of alink failure, the other link would spring into action. But STP would block allbut 1 link. So, in order to increase bandwidth, we need EtherChannel.

Configure a Layer-2 EtherChannel

Let’s create a layer-2 EtherChannel between a multilayer switch called L3 anda switch. First, a couple of things should match on the physical ports,specifically:

speed
duplex
switchport access vlan <n> and
spanning-tree cost <n>

L3(config)#int range f 0/1-2L3(config)#speed autoL3(config)#duplex autoL3(config)#mdix autoL3(config-if-range)#channel-group 1 mode ?  active     Enable LACP unconditionally  auto       Enable PAgP only if a PAgP device is detected  desirable  Enable PAgP unconditionally  on         Enable Etherchannel only  passive    Enable LACP only if a LACP device is detected

Confusingly, you configure EtherChannels from global config mode underchannel-group.

A channel-group takes a number. These numbers do not need to be the same onboth sides of the link.

We can let the devices negotiate the EtherChannel by using Cisco’s proprietaryPAgP or the standard LACP. Make sure to use the same protocol on both ends ofthe links. LACP active and PAgP desirable initiate link aggregation. LACPpassive and PAgP auto will create an EtherChannel if the other end initiatesit but will do nothing otherwise. on always creates the EtherChannelregardless of what the other end does. I do not think this should be used,though, because you might forget to configure the other end. The result will bethat you have a device that load balances over 2 links, while the other endtreats them as separate links. This might result in STP blocking a port,out-of-order packets and layer-2 loops (in short, an unstable network).

L3(config-if-range)#channel-group 1 mode activeCreating a port-channel interface Port-channel 1%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up

Verify that the port-channel got created and is working:

L3#sh ether summaryFlags:  D - down        P - in port-channel        I - stand-alone s - suspended        H - Hot-standby (LACP only)        R - Layer3      S - Layer2        U - in use      f - failed to allocate aggregator        u - unsuitable for bundling        w - waiting to be aggregated        d - default portNumber of channel-groups in use: 1Number of aggregators:           1Group  Port-channel  Protocol    Ports------+-------------+-----------+----------------------------------------------1      Po1(SD)           LACP   Fa0/1(I) Fa0/2(I)

This is not good. The Port-channel indicates the link is down. That is becausewe have not configured the other side yet. Yet another indication is thespanning-tree status, where our port-channel does not show up and instead wesee the composing interfaces Fa0/1 and Fa0/2.

#sh spanning-tree VLAN0001  Spanning tree enabled protocol ieee  Root ID    Priority    32769             Address     0001.640A.DB74             Cost        19             Port        1(FastEthernet0/1)             Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)             Address     0001.649C.A46E             Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec             Aging Time  20Interface        Role Sts Cost      Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------Fa0/1            Root FWD 19        128.1    P2pFa0/2            Altn BLK 19        128.2    P2p

Let’s configure our switch:

Switch(config)#int range f 0/1-2Switch(config-if-range)#channel-group 1 mode passiveCreating a port-channel interface Port-channel 1%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up%SYS-5-CONFIG_I: Configured from console by console%LINK-5-CHANGED: Interface Port-channel1, changed state to up%LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up

The console message inform us that the EtherChannel was created and that it is up.

Verify:

L3#sh etherchannel summary ! ... snip ...Number of channel-groups in use: 1Number of aggregators:           1Group  Port-channel  Protocol    Ports------+-------------+-----------+----------------------------------------------1      Po1(SU)           LACP   Fa0/1(P) Fa0/2(P)

If this is indeed a trunk port, we ensure that trunking is enabled.

First, configure both ends:

L3(config)#int port-channel 1L3(config-if)#switchport trunk encapsulation dot1qL3(config-if)#switchport mode trunk

Switch(config)#int port-channel 1L3(config-if)#switchport mode trunk

Notice that switchport trunk encapsulation dot1q is not necessary (or evenavailable) on a 2940 switch.

Verify both ends:

L3#sh int trunkPort        Mode         Encapsulation  Status        Native vlanPo1         on           802.1q         trunking      1

Switch#sh int trunkPort        Mode         Encapsulation  Status        Native vlanPo1         on           802.1q         trunking      1

Load Balancing

The device needs a way to determine which link to use to send PDUs over. Itcould look at the destination MAC address for example. If we have a total of 2redundant links, it looks at the last bit of the MAC address. This could beeither a 0 or a 1. If the last bit is 0, it sends it over one link, if the bitis 1, it will send it over the other. We want these bits to be as random aspossible so that both links are utilized equally. We can do this by setting theload balancing algorithm, such as src-dst-mac:

Router(config)#port-channel load-balance ?dst-ip       Dst IP Addrdst-mac      Dst Mac Addrsrc-dst-ip   Src XOR Dst IP Addrsrc-dst-mac  Src XOR Dst Mac Addrsrc-ip       Src IP Addrsrc-mac      Src Mac Addr

Run SpinRite on Librem 5 Internal Storage

Sun, 16 Feb 2025 13:34:50 +0100

I have a Purism Librem 5 Linux phone. The Evergreen batch has 32 GiB offlash storage. In this post I will show how to run SpinRite on that drive.

Since this is a rather large post, a table of contents seems in order.¹

Tools Used

A copy of SpinRite
VirtualBox (and VBoxManage on the CLI)
Wine
uuu from mfgtools

Prerequisites

I use Arch Linux, so all commands and examples are run on that OS. You should beable to easily adjust the commands to your own Unix-based OS.

I have VirtualBox version 7.1.6r167084 installed. But none of the features usedin this post are recent by any measure, so I would not be surprised if olderversions, including 6, would work fine.²

Make sure you have the appropriate VirtualBox kernel modules installed andloaded. This means at least vboxdrv:

# modprobe vboxdrv

Step 1: Buy SpinRite

It’s only $89 for the world’s best mass storage data-recovery, repair,maintenance and performance-enhancing tool.³

Step 1.1: Install Wine

To get the ISO image, we need to run the SpinRite Windows executable. To runthis executable under Linux, we need Wine. To install Wine, first make sureyou have enabled the multilib repository in /etc/pacman.conf:

[multilib]Include = /etc/pacman.d/mirrorlist

Update the package cache and install Wine:

# pacman -Sy wine

Step 1.2: Extract Bootable SpinRite ISO from SpinRite Windows Binary

Use Wine to run the SpinRite Windows executable:

$ wine /path/to/spinrite.exe

We are shown the main SpinRite installer window:

Click on “Create ISO or IMG File”:

Click on “Save a Boot Image File”:

Save the ISO somewhere you will be able to find it.

Step 2: Run SpinRite in VirtualBox

Unless you are a particularly fast typist or have all the steps scripted, theGUI is the fastest way to create a new VM. But I will show the steps andcommands for both the GUI and the CLI.

Step 2.1a: Using the GUI

Open VirtualBox. Create a new virtual machine by clicking on “New”. Give it aname of “SpinRite”. Select the stored ISO file under “ISO Image”. The type of OSshould be “Other” and the version “DOS”:

You’ll need minimal hardware to run SpinRite: choosing 1 CPU and 64 MB of RAM should suffice for version 6.1r4⁴.

Optionally, you can add a virtual hard drive to store logs:

Step 2.1b: Using the CLI

Or you can run the following commands in your terminal.

First, create the VM. The --register option registers the VM with yourinstallation of VirtualBox, instead of just creating an XML config file.

$ VBoxManage createvm --name SpinRite --ostype DOS --register

You can choose to keep the defaults for memory (128 MB) and CPU (1) or modifythem:

$ VBoxManage modifyvm SpinRite --memory 64 --cpus 1

Create a SATA controller:

$ VBoxManage storagectl SpinRite \    --name 'SATA Controller' \    --add sata

Attach the SpinRite ISO as a startup DVD:

$ VBoxManage storageattach SpinRite \    --storagectl 'SATA Controller' \    --port 0 \    --type dvddrive \    --medium /path/to/SpinRite.iso

Optionally, you can create a virtual hard drive for storing logs. --sizeis in megabytes:

$ VBoxManage createmedium disk \    --filename /path/to/SpinRite.vdi \    --size 100 \    --format VDI

Attach the virtual hard drive (on port 1 since 0 is already taken):

$ VBoxManage storageattach SpinRite \    --storagectl 'SATA Controller' \    --port 1 \    --type hdd \    --medium /path/to/SpinRite.vdi

Check your work:

$ VBoxManage showvminfo SpinRite

Step 2.2: Start the Virtual Machine

Make sure you are satisfied with the settings⁵. Then, start the VM:

$ VBoxManage startvm SpinRiteWaiting for VM "SpinRite" to power on...VM "SpinRite" has been successfully started.

You should now see the SpinRite splash screen:

Step 3: Use Jumpdrive to Access Librem 5 Internal Storage

These steps were taken from the official Librem 5 documentation.

Step 3.1: Install Jumpdrive on Your Host Machine

Download the latest build of Jumpdrive.

Extract the tar ball:

$ mkdir jumpdrive$ tar -C jumpdrive -xf purism-librem5.tar.xz

Install uuu from the AUR⁶:

$ mkdir ~/aur$ cd ~/aur$ git clone -- 'https://aur.archlinux.org/mfgtools-git.git'$ makepkg -s$ sudo pacman -U mfgtools-git-1.5.201.r3.gab8dbdf-1-x86_64.pkg.tar.zst

By the time you read this, the version of mfgtools-git will have changed.Adjust the pacman -U command accordingly.

Step 3.2: Put Librem 5 in Flash Mode

Fully power off the Librem 5.
Turn off all hardware kill switches on the Librem 5 (move all to the “bottom”position toward the bottom of the phone).
Remove the battery from the Librem 5.
Plug in the USB cable to your Linux PC.
Hold the volume-up button on the Librem 5.
Plug in the USB cable to the Librem 5. The red light should blink, and youshould not see the green light.
Reinsert the Librem 5 battery. The red light will become solid.
Release the volume-up button on the Librem 5.

If all of this seems hard, watch the video below:

Your browser does not support HTML video.

Your host Linux should recognize that a USB device has been attached.

Step 3.3: Start Librem 5 with Jumpdrive

Now, for the final step, execute:

$ ./boot-purism-librem5.sh

After a couple of seconds, you should see “Jumpdrive is running” on your phone.If you run lsblk, you should see the Librem 5 listed as a block storagedevice.

$ lsblkNAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTSsda             8:0    0 238.5G  0 disk  ├─sda1          8:1    0   500M  0 part  /boot├─sda2          8:2    0     4G  0 part  [SWAP]└─sda3          8:3    0   234G  0 part    └─cryptroot 254:0    0   234G  0 crypt /sdb             8:16   1  29.1G  0 disk  ├─sdb1          8:17   1   465M  0 part  └─sdb2          8:18   1  28.7G  0 part  sdc             8:32   1 119.3G  0 disk

In my case, the Librem 5 internal storage is available on /dev/sdb. How do Iknow this? I looked at the SIZE of the drive. Peruse the output of fdisk -l ifyou are uncertain.

Step 4: Allow Virtual Machine Access to Raw Disk Drive

Now, we need to make sure VirtualBox can work with the block device. We willcreate a disk image that points directly to the raw device.

Step 4.1: Create Medium from Raw Drive

From lsblk, we know the Librem 5 internal storage is available at /dev/sdb.SpinRite works on entire devices, so I will be ignoring partitions.

$ vboxmanage createmedium disk \    --filename=/path/to/l5flash \    --format=VMDK \    --variant=RawDisk \    --property RawDrive=/dev/sdb

This creates a virtual disk image called l5flash.vmdk in the specified path.SpinRite needs access to the raw device, so we specify --variant=RawDisk andwhere it can be found.

The output is:

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%Medium created. UUID: 541a980e-e201-44af-a927-6f3d83925bb0

We remember the UUID of the freshly-created medium.

Step 4.2: Attach Raw Disk to Virtual Machine

Attach the raw-disk medium to the virtual machine. Choose an empty port. If youdo not know, check VBoxManage showvminfo SpinRite to see which ports areavailable. In our example, ports 0 (SpinRite.iso) and 1 (logs) are alreadyoccupied, so we pick 2:

$ vboxmanage storageattach SpinRite \    --storagectl 'SATA Controller' \    --port 2 \    --type hdd \    --medium 541a980e-e201-44af-a927-6f3d83925bb0

Set --medium none if you want to detach the drive again later.

Step 5: Finally, Run SpinRite on the Drive

After starting the VM, the drive should show in SpinRite:

Incidentally, you can create a TOC from a Markdown document usingpandoc:
```
$ pandoc -f markdown -t markdown -s --toc doc.md 
```
This creates a new Markdown document that starts with a TOC. You can yank itand paste it in the original document. Make sure it makes sense first,though. ↩︎
But if it doesn’t, do let me know. ↩︎
But you know that, because you listen to Security Now! ↩︎
After executing SpinRite, a memory test is run. The accompanying texttells us it uses over 50 MB of RAM:
Taking into account the memory overhead of FreeDOS (640KB), you should begood with 64 MB. ↩︎
Personally, I like to disable things I do not use. These include NIC,audio and serial ports:
Disable network adapter 1 (and any others if present):
```
$ VBoxManage modifyvm SpinRite --nic1 none
```
Disable audio:
```
$ VBoxManage modifyvm SpinRite --audio-enabled off
```
Disable serial port COM1 (and any others if enabled)
```
$ VBoxManage modifyvm SpinRite --uart1 off
```
↩︎
On PureOS and possibly all Debian-based systems it might be in thedefault repositories: apt install uuu. ↩︎

Convert Linux into NAT Router

Sat, 15 Feb 2025 08:53:51 +0100

Sometimes it comes in handy to have a Linux box act as a router. You might nothave a switch lying around. You might want to monitor all traffic coming andgoing to some device. In this post, we will set up Linux to act as a simple IPv4NAT router.

We will call the Linux machine linux and the machine using the Linux box forinternet breakout remote.

You need 2 NICs on your Linux. They can be 2 ethernet ports. But you couldalso use your wireless interface for this. In this post, we assume they arecabled connections called eth0 and eth1.
Connect eth0 to the gateway.
Connect the two machines with an ethernet cable. The cable in the Linuxmachine should go in eth1.
Make sure the interfaces on both sides are up:
```
linux# ip link set eth1 upremote# ip link set eth0 up
```
Reminder: you can abbreviate these commands to ip l s eth<n> up.
Assign both ends an RFC 1918 private IPv4 address. You could uselink-local addresses (APIPA):
```
linux# ip addr add 169.254.0.1/32 peer 169.254.0.2 dev eth1remote# ip addr add 169.254.0.2/32 peer 169.254.0.1 dev eth0
```
Shorter: ip a a 169.254.0.1/32 peer 169.254.0.2 dev eth1
I configure the addresses to be point-to-point, because there are only 2hosts on this network and they are directly connected.
Enable IPv4 forwarding in the kernel on the Linux router
```
linux# sysctl -w net.ipv4.ip_forward=1
```
Enable NAT on the Linux router.
```
linux# iptables -t nat -A POSTROUTING \    -s 169.254.0.0/16 \    -o eth0 \    -j MASQUERADE
```
This will rewrite the source address of the packets with the original sourceaddress in the 169.254/16 network to the primary source address of theoutgoing interface eth0. The MASQUERADE target is easiest to use, but:
It should only be used with dynamically assigned IP (dialup) connections:if you have a static IP address, you should use the SNAT target.Masquerading is equivalent to specifying a mapping to the IP address of theinterface the packet is going out, but also has the effect that connectionsare forgotten when the interface goes down. This is the correct behaviorwhen the next dialup is unlikely to have the same interface address (andhence any established connections are lost anyway).
–iptables-extensions(8)
The alternative is to use the SNAT target with a static (private) IPv4address. It should be an address configured on the interface which you eitherconfigured statically yourself, or you got from a DHCP server. You can checkwith ip -br addr show eth0. Here, I am using 192.168.1.123:
```
linux# iptables -t nat -A POSTROUTING \    -s 169.254.0.0/16 \    -o eth0 \    -j SNAT --to-source 192.168.1.123
```
If you have a strict firewall on your Linux router, you might have setiptables -P FORWARD DROP. You need to allow traffic to flow between the 2NICs. For example:
```
linux# iptables -I FORWARD -i eth1 -o eth0 -j ACCEPTlinux# iptables -I FORWARD -i eth0 -o eth1 -j ACCEPT
```
Finally, make the Linux router the default gateway of the dependent box:
```
remote# ip route add default via 169.254.0.1 dev eth0
```
You can abbreviate route add to r a.

A Note on Interface Names

If you have complicated setup with virtual interfaces (VPN, VLAN) and a firewallin place, add logging to your firewall rules when packets get dropped, so youknow which interfaces to use in your source NAT (-t nat -A POSTROUTING) andFORWARD rules.

Create Server Backup from Local Machine with Borg

Fri, 14 Feb 2025 16:36:03 +0100

How to back up your servers with borg? One way to do it, as documented by thefine borg documentation, is to use a pull backup using sshfs. The ideais to mount the remote filesystem in a local directory, chroot into it, and runborg inside the chroot.

Why the chroot?

Borg stores the owner and group. Only the client (= system that gets backed up)can map from UID/GID to user and group name by looking at /etc/passwd. If wedo not chroot, then it will use the /etc/passwd of the backup server. You canbypass all this by just storing the numeric IDs always with borg create --numeric-ids

Create Backup From Remote Filesystem Using Pull

Make sure the borg repo is cryptsetuped and mounted.

Create a restricted root user on the remote system

On Debian, I need to create the /run/sshd directory first when I try to runsshd with a custom sshd_config:

# /usr/sbin/sshd -D -f /etc/ssh/sshd_config.p2p

Normally, Debian uses inetd-style Systemd sockets (seesystemd.socket(5)).

The sshd_config(5) needs to contain:

PermitRootLogin forced-commands-onlyAllowUsers root

Be careful to set PermitRootLogin to forced-commands-only:

If this option is set to forced-commands-only, root login with public keyauthentication will be allowed, but only if the command option has beenspecified (which may be useful for taking remote backups even if root login isnormally not allowed). All other authentication methods are disabled for root.

It is questionable security to allow password authentication over SSH, so let’sdisable that in sshd_config as well¹:

PasswordAuthentication no

Since we do not want to use passwords for SSH access, we create an SSH key:

$ C="$(hostname)-$USER-remotehost-remoteuser-$(date -I)"$ ssh-keygen -t ed25519 -f ~/.ssh/"$C" -C "$C"

Copy the public key to the remote host and put it in/root/.ssh/authorized_keys. Let’s also put some restrictions on this key:

restrict,command="internal-sftp" ssh-ed25519 AAA...

restrict
Enable all restrictions, i.e. disable port, agent and X11 forwarding, aswell as disabling PTY allocation and execution of ~/.ssh/rc. If anyfuture restriction capabilities are added to authorized_keys files, theywill be included in this set.
–sshd(8)

command enforces the given command (internal-sftp), even if the userprovided another command.

Unlocking the Root User

The root user might be locked by default. Locked means the user account cannotbe used. You cannot use a Unix password to log in, nor use any other means toaccess the account. Unlock root by setting the account expiry date to -1 (-e -1 = no expiry). root also needs a valid default login shell (see/etc/shells; using -s <shell> sets the shell):

# usermod -e -1 -s /bin/sh root

It is not necessary to set a password for the account nor to unlock the passwordsince we will be using SSH keys to authenticate, not passwords.

It might be a good idea to lock the account again after the backup is done:

# usermod -L -e 1 -s /usr/bin/nologin root

This disables password login, expires the account and changes the user shell tothe nologin program, which prints an error message and exits.

Make Actual Backup

Mount the remote filesystem:

$ mkdir /tmp/sshfs$ sshfs \    -o IdentifyFile=~/.ssh/<privkey> \    -p 12345 \    -o allow_other \    -o reconnect \    -o dir_cache=yes \    root@remote:/ /tmp/sshfs

-o IdentityFile should point to the file containing the private key we createdearlier.

We need -o allow_other for the root user to bind mount the borg repo into thesshfs. -o allow_other is documented in mount.fuse(5):

allow_otherThis option overrides the security measure restricting file access to theuser mounting the filesystem. So all users (including root) can access thefiles. This option is by default only allowed to root, but thisrestriction can be removed with a configuration option described in theprevious section.

Some other options to consider (from sshfs(1)):

-o reconnectautomatically reconnect to server if connection is interrupted. Attemptsto access files that were opened before the reconnection will give errorsand need to be re-opened.
-o dir_cache=BOOLEnables (yes) or disables (no) the SSHFS directory cache. The directorycache holds the names of directory entries. Enabling it allows readdir(3)system calls to be processed without network access.

Mount the borg repository inside it.

$ mkdir sshfs/mnt/borg/borgrepo# mount --bind /mnt/borg/borgrepo sshfs/mnt/borg/borgrepo

Install borg on the remote machine. Or download the borg standalone binary(the ARM version) and copy it on the remote machine. Do not forget to makeit executable:

$ cp /path/to/standalone/borg /tmp/sshfs/usr/local/bin/borg$ chmod u+x /tmp/sshfs/usr/local/bin/borg

Mount dev, proc and sys inside the sshfs²:

$ cd /tmp/sshfs$ for fs in dev proc sys; do sudo mount --bind /$fs $fs; done

Mount borg config and cache directories:

$ mkdir -p /tmp/sshfs/root/.config/borg$ chmod 0700 /tmp/sshfs/root/.config/borg$ sudo mount --bind /root/.config/borg /tmp/sshfs/root/.config/borg$ mkdir -p /tmp/sshfs/root/.cache/borg$ chmod 0700 /tmp/sshfs/root/.cache/borg$ sudo mount --bind /root/.cache/borg /tmp/sshfs/root/.cache/borg

Finally, enter the chroot:

# chroot /tmp/sshfs

Things to Pay Attention to

Do Not Back Up borg Repository

When running borg create, make sure to --exclude /mnt/borg/borgrepo.

Do Not Use Inode Numbers for borg Caching

Set --files-cache ctime,size. See borg-create(1). The reason is that,by default, borg uses ctime,size,inode to determine whether a file has beenchanged. But sshfs cannot guarantee stable inode numbers. If the inodenumbers are not stable, the cache is useless: all files look like they havechanged from invocation to invocation.

Provide borg Passphrase over Anonymous Pipe

If you do not want to type in your credentials when prompted, you can pass inyour credentials with pass (or another program that outputs the secret over ananonymous pipe):

# pass show borg-secret | \    chroot /tmp/sshfs \    BORG_PASSPHRASE_FD=0 borg create repo::archive /home /etc

BORG_PASSPHRASE_FD takes a file descriptor number. 0 is stdin. Seeborg(1).

Tear Down

After the backup is done, exit the chroot and unmount all mounted filesystemsrecursively:

chroot# exitlocal# umount -R /tmp/sshfs

This Was Tested on the Following Versions

$ borg --versionborg 1.4.0$ sshfs --versionSSHFS version 3.7.3FUSE library version 3.16.2using FUSE kernel interface version 7.38fusermount3 version: 3.16.2

This means that even when you accidentally set PermitRootLogin to yes,you still cannot log in using a password, because PasswordAuthenticationtakes precedence. ↩︎
Why mount /dev/, /proc and /sys? /dev provides device files that are essential for programs inside the chroot to interact with hardware, such as terminals, disks, and other hardware resources. See also [hier(7)`]12 and the Wikipedia entry ondevice files.
/proc is a virtual filesystem that provides information about runningprocesses and the kernel. Programs use /proc to gather system information,monitor processes, and manage resources. See also hier(7),proc(5) and the Wikipedia entry on procfs.
/sys provides an interface to the kernel’s internal data structures. Itallows programs to access and modify kernel parameters, device drivers, andother system settings. Mounting /sys enables programs within the chroot tointeract with the kernel. See also hier(7) and the Wikipedia entryon sysfs.
But this is all theory. If you do not mount these filesystems, borg willstill run fine. Maybe I will dive into this in another blog post: how do youknow what you need in a chroot? ↩︎

Updating My Cisco 4321 Router's IOS over HTTP, FTP and TFTP and More

Wed, 12 Feb 2025 16:38:28 +0100

I recently got my hands on a second-hand Cisco 4321 router. First things first,I want to upgrade to the latest IOS version so I have the latest securitypatches. Having no experience at all with IOS, these are some things I learnedalong the way.

Conventions Used in This Post

Router> is a command prompt that indicates the user in currently in EXEC mode.Router# indicates that the user is in privileged EXEC mode. You get from EXECto privileged EXEC mode by:

R1>enableR1#disableR1>

Depending on your setup, you might need to type a password after enable.

Every owner of Cisco equipment knows this, but I am one of my own futurereaders, so I need to keep things simple.

What Version Do I Currently Run?

R1>sh version

That shows:

Cisco IOS XE Software, Version 16.06.04Cisco IOS Software [Everest], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.4, RELEASE SOFTWARE (fc3)...System returned to ROM by Reload CommandSystem image file is "bootflash:isr4300-universalk9.16.06.04.SPA.bin"Last reload reason: Reload Command...

I am running IOS XE 16.06.04 and it got loaded from the fileisr4300-universalk9.16.06.04.SPA.bin stored on the bootflash.

Finding an Appropriate IOS Image

Turns out Cisco are ~~dicks~~ difficult about updates. You need a service contractbefore you can get a download link.

You can find your router model and navigate to the downloads tab. If youhave a support contract, you can log in and presumably download the latestversion. Also, Cisco hints that some older version might be available withouta contract. But why would you want an old version with lots of security problemsto run on an internet-facing router, right?

I do not have a service contract. I did notice the previous owner had a newerversion of the firmware on the router’s flash memory:

R1#dir :flash...18  -rw-  763905745  Nov 9 2023 15:45:02 +00:00 isr4300-universalk9.17.09.04.SPA.bin...

Also, it looks like sometimes an IOS release falls off a truck. Search yourfavorite search engine for something like intitle:index.of <image-name> to get a sense of where these trucks drive.

Making Sure You Did Not Just Download Malware

How do I know the firmware I got was not tampered with? Get the MD5 and SHA512checksums from the Cisco Software Download page when hovering over thefilename:

At least they did not hide that information.

Now, let’s verify if the checksum on my router equals to what was published. MD5is old and broken, so let’s go with SHA512:

R1#verify /sha512 flash:isr4300-universalk9.17.09.04.SPA.bin 34f8eff288bb485c8ca902d2e5c03f9d45a4dc881e919a6ae3f0ca3a52161eecc5c9fc7a7ed90423984ee690695293ad011590fa0e75c65ccf003f7f89b0d5c0........................................................ <snip> ...Done!Verified (bootflash:isr4300-universalk9.17.09.04.SPA.bin) = 34f8eff288bb485c8ca902d2e5c03f9d45a4dc881e919a6ae3f0ca3a52161eecc5c9fc7a7ed90423984ee690695293ad011590fa0e75c65ccf003f7f89b0d5c0

verify /sha512 flash:<file> calculates the checksum for the specified file. Ifyou provide a checksum yourself with verify /sha512 flash:<file> <checksum>,the checksum is verified against your input. In this case, the checksum matches.Sweet!

Looking at the image name, I can also see that it contains SPA. According toCisco, that stands for an image Signed with key version A meant forProduction use. The file can be verified as follows:

R1#verify flash:isr4300-universalk9.17.09.04.SPA.binVerifying file integrity of bootflash:isr4300-universalk9.17.12.04a.SPA.bin...Embedded Hash   SHA1 : 03A19E90D4E58FEDCAC30959192B8CEEDF6D29CCComputed Hash   SHA1 : 03A19E90D4E58FEDCAC30959192B8CEEDF6D29CCStarting image verificationHash Computation:    100%Done!Computed Hash   SHA2: 975953771aac683df79da71226460ecb                      fead063cf39b9b4f4165e8581dee89fd                      733868cc148e407e76ace6c2f912dac9                      fb7c95bdbd402b3fffa257c9ba6ebbd4                      Embedded Hash   SHA2: 975953771aac683df79da71226460ecb                      fead063cf39b9b4f4165e8581dee89fd                      733868cc148e407e76ace6c2f912dac9                      fb7c95bdbd402b3fffa257c9ba6ebbd4                      Digital signature successfully verified in file bootflash:isr4300-universalk9.17.12.04a.SPA.bin

Then there is the following command that:

Displays software authenticity-related information for the current ROMmon andthe Cisco IOS image file used for booting.

R1#sh software authenticity running SYSTEM IMAGE                                ------------                          Image type                    : Production      Signer Information                          Common Name           : CiscoSystems        Organization Unit     : IOS-XE              Organization Name     : CiscoSystems    Certificate Serial Number : 675E97D5    Hash Algorithm            : SHA512    Signature Algorithm       : 2048-bit RSA    Key Version               : A                                                              ROMMON------... snip ...

You can view more information about the public keys used to verify the signedimages:

R1#show software authenticity keys

At this point, I am not sure where my router got these public keys from, so I amhappy to just verify the checksum of the firmware image I am downloading on therouter. But good to know that there exist more robust ways to verify thefirmware actually came from Cisco.

What Transfer Methods Are Available?

R1>enableR1#copy ?  /erase          Erase destination file system.  /error          Allow to copy error file.  /noverify       Don't verify image signature before reload.  /verify         Verify image signature before reload.  bootflash:      Copy from bootflash: file system  cns:            Copy from cns: file system  flash:          Copy from flash: file system  ftp:            Copy from ftp: file system  http:           Copy from http: file system  https:          Copy from https: file system  null:           Copy from null: file system  nvram:          Copy from nvram: file system  rcp:            Copy from rcp: file system  running-config  Copy from current system configuration  scp:            Copy from scp: file system  startup-config  Copy from startup configuration  system:         Copy from system: file system  tar:            Copy from tar: file system  tftp:           Copy from tftp: file system  tmpsys:         Copy from tmpsys: file system  usb0:           Copy from usb0: file system  webui:          Copy from webui: file system

Most of these options look familiar, but some I have never heard of, such ascns: and webui:.

Sneakernet (USB)

My router has a USB A port. When I plug in a USB drive (single partitionformatted as FAT32), it shows up under usb0. Copy the image to flash memory:

R1#copy usb0:image.bin flash:

But you might not always have a USB drive handy (or a USB port for that matter),so let’s look at some other ways to transfer files.

Using IP

Transfer methods using IP should first have IP configured on the router:

R1>enableR1#conf tR1(config)#int g0/0/0R1(config-if)#no shutR1(config-if)#ip addr 10.0.0.1 255.255.255.252R1(config-if)#end

On the other side, configure 10.0.0.2. I am assuming Linux:

# ip addr add 10.0.0.2/32 peer 10.0.0.1 dev eth1

Using SCP to Pull and Push Files

Make sure SSH runs and you have a user configured with privilege level 15. Then,enable the SCP server on IOS:

R1(config)#ip scp server enable

Now we can use scp(1) to pull and push files. Notice, though, that inrecent times:

scp uses the SFTP protocol over a ssh(1) connection for data transfer, anduses the same authentication and provides the same security as a loginsession.

You can enable the legacy behavior with the -O switch.

To pull a file called A_FILENAME from IOS:

linux$ scp -O stefan@10.1.1.1:flash:A_FILENAME ./target-filename

If you get the error “protocol error: filename does not match request”, youcould disable strict checking of filenames with the -T flag.

To push a file called A_FILENAME to IOS:

linux$ scp -O ./A_FILENAME stefan@10.1.1.1:flash:A_FILENAME

Pulling a File over HTTP

Cisco IOS can reach out to an HTTP(S) server and pull a file from it. I like touse Busybox’ httpd for this:

linux$ busybox httpd -vv -f -p 5000

R1#copy http://10.0.0.2:5000/image.img flash:

10.0.0.1:13911: url:/file.txt10.0.0.1:13911: response:20010.0.0.1:26393: url:/file.txt10.0.0.1:26393: response:200

Notice the duplicate requests coming from the router.¹

Pushing a File over HTTP

IOS can also run an HTTP(S) server and accept file uploads. For simplicity’ssake, I am only demonstrating how to configure file uploads over HTTP. Runningan HTTPS server is discussed in another post.

First, prepare the device to receive file over HTTP:

ISR4321# mkdir flash:uploadsISR4321# conf tISR4321(config)# ip http serverISR4321(config)# ip http upload enable path flash:uploads

$ curl -T isr4300.priv.pem \192.168.88.254/flash:uploads/isr4300.priv.pem

curl(1)’s -T, --upload-file <file> switch in combination with anHTTP URL will PUT the file on the remote destination. If you end the URL with aforward slash /, curl(1) will append the local filename to the URL.

If you want to amend the filename on the Cisco file system:

ISR4321# rename flash:uploads/incorrect.name flash:uploads/correct.name

TFTP

linux# busybox udpsvd -vE 10.0.0.2 69 busybox tftpd -u ftp /srv/ftp

Busybox’ tftpd is meant to be run by inetd. That daemon would bindto a port, and when a connection is made to that port, it spawns a process whichruns an executable. It hooks the network stream directly to stdin and stdout ofthe process. A big upside is that the process itself does not need to have anynetwork code in it.

Here, instead of starting inetd, we take Busybox’ suggestion and run the helperprogram called udpsvd that works much like inetd but runs as a foregroundprocess:

Create UDP socket, bind to IP:PORT and wait for incoming packets. Run PROGfor each packet, redirecting all further packets with same peer ip:port to it.

We need to run as root to be able to bind to privileged port 69. tftpd alsostarts as root to be able to chroot into the specified directory, but thendrops its privileges to access the files as -u <user>.

On the router we can now transfer files:

R1#copy tftp://10.0.0.2/file.txt flash:Destination filename [file.txt]?Accessing tftp://10.0.0.2/file.txt...Loading file.txt from 10.0.0.2 (via GigabitEthernet0/0/0): ![OK - 17 bytes]17 bytes copied in 0.018 secs (944 bytes/sec)

Also, depending on the TFTP software you are using, you might run into thiscaveat:

Most TFTP applications cannot transfer files larger than 16MB in size. If theCisco IOS software you install is larger than 16MB, you must use an FTP or RCPserver.

I did not have this problem with busybox tftpd, though.

A Cisco router can also function as a TFTP server, not only as a client. Toenable the TFTP server and serve a file called A_FILENAME:

R1(config)#tftp-server flash:A_FILENAME

We can now use an TFTP client to download the file:

linux$ busybox tftp -g -r A_FILENAME 10.1.1.1

FTP

Transferring files over FTP works much the same as TFTP. Instead of using UDP,it uses TCP, though, so we have to change the Busybox’ helper program:

linux# busybox tcpsvd -vE 10.0.0.2 21 busybox ftpd -vv -a ftp /srv/ftp

On the router:

R1#copy ftp://10.0.0.2/file.txt flash:Destination filename [file.txt]?Accessing ftp://10.0.0.2/file.txt...Loading file.txt[OK - 17/4096 bytes]17 bytes copied in 0.048 secs (354 bytes/sec)

Over a Serial Connection With `minicom`

On the internet, I read that in ROMMON, you can use something called “xmodem” totransfer files. minicom has a “send files” feature where you can choose thexmodem protocol.

It should work like:

rommon 123 >copy xmodem: flash:image.img

However, after entering ROMMON by booting up the router and pressingCtrl+A and then F to send a BREAK, I see onlythe following options:

rommon 1 > helpalias               set and display aliases commandboot                boot up an external processconfreg             configuration register utilitydev                 list the device tabledir                 list files in file systemhelp                monitor builtin command helphistory             monitor command historymeminfo             main memory informationrepeat              repeat a monitor commandreset               system resetset                 display the monitor variablesshowmon             display currently selected ROM monitorsync                write monitor environment to NVRAMtoken               display board's unique token identifierunalias             unset an aliasunset               unset a monitor variable

Note there is no copy command. (Nor an tftpdlnd command I saw mentionedelsewhere. That command should, after setting the appropriate environmentvariables, download files over TFTP.)

I am not sure at this point if this feature is supposed to be supported by everyCisco router or not. Maybe I can unlock a privileged mode (akin to enable inIOS) or update the ROMMON firmware and get extended options? It would suck forsure if all you had was ROMMON mode and no way to boot an image or get a workingimage in persistent memory.

Speed Considerations

TFTP is slow, because it uses UDP segments with a payload of 512 bytes each.It is a “lockstep” protocol: each data packet needs to be acknowledged bythe receiver before the sender sends the next packet.
FTP and HTTP use TCP. TCP allows for more payload per segment and manysegments can be outstanding before the sender stops to wait for anacknowledgement. The receiver can acknowledge many segments at once, reducingoverhead. The receive window size (amount of bytes that can be outstanding atany one time before the sender has to wait for acknowledgement) can grow tomany megabytes.
Sneakernet should be even faster. If the router supports USB 2.0, you shouldbe able to expect around 30+ MiB/s depending on factors such as how fast therouter can flush the data to persistent storage.

For example, the transfer of a 700+ MiB image was so tediously long over TFTP, Ihad to abort. FTP, on the other hand, completed in less than 5 minutes:

776523079 bytes copied in 268.943 secs (2887315 bytes/sec)

Security Considerations

Protocols like HTTP, FTP and TFTP transmit data in plaintext and do not provideany protections against snooping and tampering. While we might not care muchabout confidentiality when transmitting a router image, we definitely want tomake sure we connect to the correct download server and get the correct imageloaded in our router. Accidental alterations during transmission, such as bitflips, are caught by transport layer (UDP/TCP) checksums: the faulty UDP/TCPsegment is discarded and the download server resends it automatically. Maliciousalterations by a MITM can be detected by calculating the digest of thereassembled file on the destination:

On your Linux download server:

linux$ sha512sum isr4300-universalk9.17.09.04.SPA.bin34f8eff288bb485c8ca902d2e5c03f9d45a4dc881e919a6ae3f0ca3a52161eecc5c9fc7a7ed90423984ee690695293ad011590fa0e75c65ccf003f7f89b0d5c0  isr4300-universalk9.17.09.04.SPA.bin

Compare that output with what you get on the router:

R1#verify /sha512 flash:isr4300-universalk9.17.09.04.SPA.bin 34f8eff288bb485c8ca902d2e5c03f9d45a4dc881e919a6ae3f0ca3a52161eecc5c9fc7a7ed90423984ee690695293ad011590fa0e75c65ccf003f7f89b0d5c0........................................................ <snip> ...Done!Verified (bootflash:isr4300-universalk9.17.09.04.SPA.bin) = 34f8eff288bb485c8ca902d2e5c03f9d45a4dc881e919a6ae3f0ca3a52161eecc5c9fc7a7ed90423984ee690695293ad011590fa0e75c65ccf003f7f89b0d5c0

To trust this digest, you need a secure connection to the router, however. Thiscan be a connection via the serial console or over the LAN, for example, or overSSH. If you do not have a secure connection to the destination, you cannot besure and maybe should not be servicing your router this way. Spend time to setup SSH in this case.

Also, as mentioned at the top,if the image has something like SPA in it, it should be digitally signed.I have not spent much time looking into this, but if you can be certain you havethe correct public key from Cisco, this would provide the strongest guaranteethat the image has not been tampered with.

Specifying Which Image to Run on Startup

R1#conf tR1(config)#boot system flash isr4300-universalk9.17.12.04a.SPA.binR1(config)#endR1#copy running-config startup-configR1#reload

Make sure to persist changes by writing the config to flash memory.

That is why I could not get Busybox’ nc working:
```
linux$ cat -A res.http#!/bin/shcat <<EOFHTTP/1.1 200 OK^M$Content-Type: text/plain^M$Content-Length: 14^M$^M$Hello, world!$EOF
```
```
linux$ chmod u+x res.http
```
```
linux$ busybox nc -lk -p 5000 -e ./res.httpconnect to 10.0.0.2:5000 from 10.0.0.1:16289 (10.0.0.1:16289)
```
Wireshark shows nc sending TCP RSTs after the first HTTP exchange with therouter, whereas maybe the router only tolerates graceful connection termination,over which we have no control when using nc. (Same goes for using busybox tcpsvd which I will discuss later.)
Then again, maybe these duplicate HTTP requests are just an idiosyncrasy ofisr4300-universalk9.16.06.04.SPA.bin, the version I am trying to upgrade from. ↩︎

VLANs

Tue, 11 Feb 2025 11:27:00 +0100

VLANs have stumped me for the longest time. They are a requirement for networksegmentation, but couldn’t you also achieve that by subnetting? What is therelationship between a VLAN and a subnet? Are they the same thing? In this post,I attempt to answer these and some other questions.

The LAN

A LAN is a layer-2 broadcast segment. That means that layer-2 devices, such asswitches, will flood broadcasts, unknown multicasts and multicasts (BUM) outof every interface except the one they received it on. Every NIC on every deviceon that same segment will have to spend CPU cycles examining the frame to see ifit is addressed to that NIC.

In the diagram, if host A sends a BUM to its LAN (the orange arrow), the switchwill forward it out all of its other ports.

To create a second, separate broadcast domain, we need to buy a second switch(and not connect the switches together):

Now, when host A sends traffic onto the link facing the switch, the BUM andunicast traffic will be flooded onto the segment connected with host B.Unsurprisingly, since the two switches are not connected, no traffic coming fromhost A or B will ever reach host C and D.

We have achieved traffic isolation, but at the price of being potentially verywasteful: if you have a 24-port switch and require 12 devices on 2 different LANsegments, you have to buy 2 switches, even though the single switch has enoughports to spare.

Why Segment Traffic

But let’s back up a little and think about why we might want to segment trafficat all:

We might want to limit CPU cycles spent by NICs on all BUM traffic.
We might not want snoopers to have access to all BUM traffic.
We might want to apply different security policies to different kinds ofdevices and/or traffic (think network management traffic such as SSH bytrusted IT personel versus guest Wi-Fi).
We might want people in different physical locations be part of the samebroadcast domain.
We might want to keep the STP tree simple and manageable.

Side Note: Why Not Just Route Everything?

When I was building my first networks, I thought, why not just use routing? Thatis, why not just put every device on its own IP subnet and route all trafficbetween devices?

While that definitely works, because routing also gives us traffic isolation,it carries more overhead: a router needs to examine the IP header and changeit (TTL, recalculate checksum), whereas a switch looks up the destinationaddress in its CAM table and immediately forwards the frame. Switching, in aword, is faster (wire speed) than routing (10-50+ μs per hop).
Also, switching requires less or no configuration: you plug the switch in,power it up, attach devices and they can talk, all without setting up(additional) IP subnets.
Also, routers (and L3 switches) are more expensive and generally have lessports.

Introducing VLANs

VLANs are virtual broadcast domains. They are identified by a 12-bit number(0-4095, although 0 and 4094 and some others are reserved) that groups BUMtraffic together. Traffic flowing in different VLANs is completely isolated, asif we had 2 separate, unconnected switches:

In the diagram, traffic from host A can only ever reach host B or be discarded.VLANs guarantee that no traffic will ever flow from host A or B to host C or D,just as if the switches were physically separated.

To make all traffic coming in on a link part of a VLAN, we can make that portpart of the VLAN by assigning it a VLAN ID.

To pass VLAN traffic from switch to switch, we need to somehow tell the otherswitch which VLAN the traffic belongs to. The easiest approach is to use a PortVLAN ID (PVID or access port) on both sides and use a link per VLAN:

Obviously, that doesn’t scale very well. You would need a separate link forevery VLAN. Image having not 2, but 10 or 30 VLANs and soon most of the ports ofour switches are unavailable for end devices (the reason we have these switchesin the first place).

Enter trunks: we use a single link between switches but we embed the VLAN IDinside the frame:

This approach has been standardized in the IEEE 802.1Q standard:

The 802.1Q field is inserted before the EtherType field and has type 0x8100itself. The receiving switch strips the additional field from the Ethernetheader and forwards it towards its destination.

Routing Traffic between VLANs

Traffic being completely isolated from each other is great, except when you needto talk to a device in another VLAN. To do so, a layer-3 device, such as arouter or a layer-3 switch, is needed. Every VLAN typically gets assigned itsown subnet, and routers route between these subnets and thus between the VLANs.

One of the most typical topologies for routing between VLANs is the so-calledRouter-on-a-Stick (ROAS):

Here, a switch that knows about several VLANs is connected with a single trunkport to a router. The router uses virtual subinterfaces that each know about aseparate VLAN. Those subinterfaces are assigned IP addresses in the VLAN subnet.Upon receiving traffic that is part of a specific VLAN, the router decapsulatesthe Ethernet frame, inspects the IP header, selects the outgoing interface,encapsulates the packet into a new Ethernet frame including an 802.1Q field, andsends it out of the virtual subinterface (over the single physical trunk port)back to the switch.

Configure Cisco

On a Cisco switch, configure an access port as follows:

Switch(config)#int gi 0/1Switch(config-if)#switchport mode accessSwitch(config-if)#switchport access vlan 10

Verify:

Switch#show vlan

To create a trunk port:

Switch(config-if)#switchport encapsulation dot1qSwitch(config-if)#switchport mode trunk

To allow which tags are allowed on the trunk:

Switch(config-if)#switchport trunk allowed vlan 10,20

Verify:

Switch#show interface trunk

Configure MikroTik

Throughput Disclaimer

On MikroTik devices, things are vastly more configurable, more complex andtherefore more confusing. First, there is no single configuration that works onall devices. Some switches have specialized hardware (docs here andhere) and the configuration reflects that. You can increase throughput bymaking use of hardware processing instead of using the CPU. While I’ve found theconfig below to work on a couple of MikroTik devices including a CRS328 L3switch, I cannot guarantee that it is the best way to do it on all hardware.Read the documentation.

Bridges, Bridge Ports, VLAN interfaces and Tagged and Untagged Egress

With that out of the way, here, I am going to configure a hEX router thathas 5 ports, ether1 through ether5. ether3 and ether4 are going to beaccess VLANs, ether5 is going to be a trunk port.

First you create a bridge:

[u@hEX] /interface/bridge/add name=BR[u@hEX] /interface/bridge/print proplist=name,pvid,vlan-filteringFlags: D - dynamic; X - disabled, R - running  0  R name="BR1" pvid=1 vlan-filtering=no

Notice that vlan-filtering says no for the time being to ease configuration.

Add VLAN interfaces:

[u@hEX] /interface/vlan/add name=GUEST_WIFI vlan-id=10 interface=BR[u@hEX] /interface/vlan/add name=MGMT vlan-id=20 interface=BR

Optionally add IP addresses to those VLAN interfaces so that the MikroTik can bereached on those IPs.

Add ports to the bridge and make ether3 and ether4 access ports by settingtheir PVID:

[u@hEX] /interface/bridge/port/add bridge=BR interface=ether3 pvid=10[u@hEX] /interface/bridge/port/add bridge=BR interface=ether4 pvid=20[u@hEX] /interface/bridge/port/add bridge=BR interface=ether5

Now for the interesting part: we need to determine at which point duringits journey from the physical interface, over the bridge to the VLAN interfacean incoming packet should be tagged

I am not completely sure, but it seems to me that a packet should always have atag on the bridge. How else could the bridge determine to what VLAN interface tosend it? Access ports should send traffic untagged on egress, trunk ports shouldsend traffic tagged. And so we end up with the following configuration foraccess ports 3 and 4:

[u@hEX] /interface/bridge/vlan/add bridge=BR vlan=10 \\... tagged=BR untagged=ether3[u@hEX] /interface/bridge/vlan/add bridge=BR vlan=20 \\... tagged=BR untagged=ether4

Traffic on the trunk port should be tagged both on the bridge and the egressinterface:

[u@hEX] /interface/bridge/vlan/add bridge=BR vlan=10,20 \\... tagged=BR,ether5

Finally, don’t forget to enable PVID behavior and VLAN filtering:

[u@hEX] /interface/bridge/set [find name=BR] vlan-filtering=yes

Configure Linux

Linux is the underlying OS for both Cisco’s IOS XE and MikroTik’s RouterOSso it should surprise no-one that you can configure everything in this articleon Linux as well.

However, the one thing I found to be useful, is tagging my Linux traffic with802.1Q tags. To do that, create a subinterface and configure the tag:

linux# ip link add link eth0 name eth0.10 type vlan id 10linux# ip address add 10.10.10.2/24 dev eth0.10linux# ip link set eth0.10 up

How to Enable a DHCP Client on a Cisco Interface

Tue, 11 Feb 2025 10:55:07 +0100

Let’s look at how to automatically get an IP addresses assigned on an interfaceon a Cisco device. This could not be simpler.

To make interface g0/0/0 a DHCP client:

ISR4321(config)# int g0/0/0ISR4321(config-if)# ip address dhcpISR4321(config-if)# no shutdown

If there is a DHCP server on the network, the interfaces DHCP Discover probeswill be honored and the interface will now have an IP address:

ISR4321# sh ip int briefInterface             IP-Address      OK?  Method  Status  ProtocolGigabitEthernet0/0/0  192.168.88.254  YES  DHCP    up      upGigabitEthernet0/0/1  unassigned      YES  unset   down    downGigabitEthernet0      unassigned      YES  unset   down    down

Authentication on Cisco IOS

Tue, 11 Feb 2025 08:35:09 +0100

Let’s have a look at how Cisco IOS handles authentication and how passwords arestored in the configuration file.

Disable Cleartext Passwords in the Configuration File

Let’s I set the “front-door password” for console access that would put the userin user EXEC mode:

ISR4321# conf tISR4321(config)# line console 0ISR4321(config-line)# password foobarbazISR4321(config-line)# loginISR4321(config-line)# ^Z

ISR4321# sh run | b lineline con 0!line aux 0!line vty 0 4 password foobarbaz loginline vty 5 15 password foobarbaz login!

To obfuscate passwords in the configuration file:

ISR4321(config)# service password-encryption

Passwords that are normally be stored in plaintext in the config file will nowbe obscured by “Type 7 encryption”.¹ “Encryption” sounds like a big deal, butthis type of encryption is merely meant to prevent casual shoulder surfing:

line con 0 password 7 1047021200 login

Type 7 “encryption” is seriously broken. So consider anyone with access tothe config (or backups thereof) to know the password and be able to access userEXEC mode on your Cisco devices.

If we were to exit now, we would be prompted for this front-door password:

ISR4321# exitISR4321 con0 is now availablePress RETURN to get started.User Access VerificationPassword: ISR4321>

The router does not echo the password back to the screen to prevent shouldersurfing.

Protect Privileged EXEC Mode with a Password

Set the “enable” password (to get to enable/privileged EXEC mode):

ISR4321(config)# enable secret algorithm-type scrypt secret <pwd>

At least on my IOS, fortunately, scrypt is available as a PBKDF.²

When enable secret is set, IOS will ignore the enable password. That meansthat if both are set, you can only gain access with the enable secretpassword. enable password stores the password in plaintext in the config file,or merely obfuscates it when service password-encryption is set. So removethis insecure password with no enable password.

Username + Password Authentication

You can also use username + password authentication. Logins over console andTelnet can then request a username in addition to a password. SSH alwaysrequires both a username and a password.

Create users with usernames, password and privilege levels:

ISR4321(config)# username stefan privilige 15 secret stefanpwdISR4321(config)# username thomas privilige 1 secret stefanpwd

You can use algorithm-type to specify the PBKDF. On my IOS, by default, ituses the (weak) MD5. Use scrypt instead.

By changing line console 0 to use login local instead of login, the loginprompt will request credentials from the local list configured with username <name> secret <secret>:

ISR4321# conf tISR4321(config)# line console 0ISR4321(config-line)# login localISR4321(config-line)# endISR4321# exitUser Access VerificationUsername: thomasPassword:ISR4321>

If we authenticate with the privileged user stefan instead (which wasconfigured with privilege 15), we are taken straight to enable/privileged EXECmode after authentication.

Secure SSH and/or Telnet with ACLs

To secure logging in over IP, ACLs are needed. After all, you can have a strongpassword, but if Cisco’s SSH or Telnet daemon has a vulnerability, you wouldstill want to prevent the baddies from knocking on those exposed ports.

First, create an standard numbered ACL:

ISR4321(config)#access-list 1 permit 172.16.0.0 0.15.255.255

This permits all hosts with a source IP address in the RFC 1918 private IPsubnet 172.16/12. ACLs in Cisco land always end with default deny. So thisreads: allow IP packets from source 172.16/12 and drop the packets otherwise.The wildcard mask (0.15.255.255) is a mask that defines which bits of thesource address will be checked against the defined ACL base address.

Say, a packet with source IP 172.16.123.123 comes in:

base  = 1010 1100 0001 0000 0000 0000 0000 0000mask  = 0000 0000 0000 1111 1111 1111 1111 1111check = 1010 1100 0001 0000 0111 1011 0111 1011

If we compare base with check for every bit where the mask has a 0 bit,we see that base == check. And so the ACL matches and traffic is allowed toflow. Internally, Cisco probably just logically ORs the base and mask (base | mask) and mask and check (mask | check) and compares the results.

Currently, this ACL is not assigned to a port or a vty though, so it does not doanything.

ISR4321(config)#line vty 0 15ISR4321(config-line)#access-class 1 in

Here, we assign the standard numbered ACL with identifier 1 to all the vtylines.

The in keyword specifies the direction of the traffic flow: in this case, weanticipate a remote client initiating a connection with the Cisco device. Youcould also specify out. The ACL would then filter outgoing connection madefrom the Cisco device by the user connected over a vty line. The single IPaddress in the standard ACLs is then interpreted as a destination rather thana source IP address. When using extended ACLs (100-199 and 2000-2699) foraccess-class, you should use the any keyword for either the source ordestination address, depending on whether ingress or egress is filtered.

Note that if you disable password-encryption again, IOS will not decryptthe obfuscated passwords. ↩︎
By default, my IOS version (16.06.04) uses the (weak) MD5-based BSDpassword algorithm 1. If you encounter an enable secret:
```
Router#sh run | i secretenable secret 5 $1$mERr$jegsBINbJe1/UUQKc.NUC1
```
You can check the password on a Linux box with openssl-passwd(1):
```
openssl passwd -1 -salt mERr foobarbaz$1$mERr$jegsBINbJe1/UUQKc.NUC1
```
↩︎

Set the Time Manually or with NTP on Cisco IOS

Tue, 11 Feb 2025 08:20:46 +0100

Let’s take a look at how to set the clock on a Cisco device.

Display the Current Time

ISR4321> show clock13:11:59.001 UTC Sun Mar 2 2025

Manually Set the Time

ISR4321# clock set 13:08:10 2 March 2025

Configure NTP

ISR4321(config)# ntp server 192.168.88.1

Set timezone and DST:

ISR4321# conf tISR4321(config)# clock timezone CET +1ISR4321(config)# clock summer-time CEST recurring last Sunday March 02:00 last Sunday October 03:00ISR4321(config)# end

After recurring, you can indicate when DST time starts. In this case, it readssomething like “summer time starts the last Sunday of March at 02:00 in themorning; winter time starts the last Sunday of October at 03:00 in the morning.”

Make Cisco Device Start Up From NVRAM Again

Tue, 11 Feb 2025 08:05:13 +0100

My ISR4321 was not remembering its config on startup even though I was sure towrite memory after making config changes. Turns out, the configurationregistry was not set correctly. This post looks at what this registry containsand how to make sure it will load the stored config on bootup.

The configuration registry is 2 bytes represented as 4 hex digits. In my case,the it was set to 0x9922.

Here is an overview of what all the different bits mean. When breaking itdown:

0x8000 => enables diagnostic messages, ignores NVRAM contents0x1000 => console line speed0x0800 => console line speed0x0100 => break disabled0x0020 => console line speed0x0002 => boots into ROM if inital boot fails

These values are ANDed. The first value, 0x8000, causes IOS to ignore thestartup-config from NVRAM. We change that value:

ISR4321# conf tISR4321(config)# config-register 0x3922

The default is 0x2102, but I want the fasted baud rate (115200) so I add0x1820 to get 0x3922.

Create or Import TLS Certificates on Cisco IOS

Tue, 11 Feb 2025 07:28:44 +0100

You can generate a self-signed certificate on Cisco IOS. If that fails, youcan generate one on another machine and import it.

Let IOS Generate a Self-Signed Certificate

First, we create a CA:

ISR4321# conf tISR4321(config)# crypto pki trustpoint mytpISR4321(ca-trustpoint)# enrollment selfsignedISR4321(ca-trustpoint)# fqdn isr4300.testISR4321(ca-trustpoint)# ip-address 192.168.88.254ISR4321(ca-trustpoint)# subject-alt-name isr4300.testISR4321(ca-trustpoint)# hash sha256

Now, we can create a self-signed TLS certificate:

ISR4321(config)# crypto pki enroll mytpMar  4 15:28:56.870: %CRYPTO-6-AUTOGEN: Generated new 1024 bit key pair% Include the router serial number in the subject name? [yes/no]: noGenerate Self Signed ISR4321 Certificate? [yes/no]: yesISR4321 Self Signed Certificate successfully created

Inspect the just-created certificate details:

ISR4321# sh crypto pki certificates mytpISR4321 Self-Signed Certificate  Status: Available  Certificate Serial Number (hex): 02  Certificate Usage: General Purpose  Issuer:    serialNumber=FDO2431M1QP+hostname=ISR4300+ipaddress=192.168.88.254  Subject:    Name: ISR4300    IP Address: 192.168.88.254    Serial Number: FDO2431M1QP    serialNumber=FDO2431M1QP+hostname=ISR4300+ipaddress=192.168.88.254  Validity Date:    start date: 16:29:19 CET Mar 4 2019    end   date: 01:00:00 CET Jan 1 2020  Associated Trustpoints: mytp

Export the certificate:

ISR4321# conf tISR4321(config)# crypto pki export mytp pem terminal% Self-signed CA certificate:-----BEGIN CERTIFICATE-----MIICWTCCAcKgAwIBAgIBAjANBgkqhkiG9w0BAQUFADBIMUYwEgYDVQQFEwtGRE8yNDMxTTFRUDATBgkqhkiG9w0BCQIWBlJvdXRlcjAbBgkqhkiG9w0BCQgTDjE5Mi4xNjguODguMjU0MB4XDTE5MDMwNDE1MjkxOVoXDTIwMDEwMTAwMDAwMFowSDFGMBIGA1UEBRMLRkRPMjQzMU0xUVAwEwYJKoZIhvcNAQkCFgZSb3V0ZXIwGwYJKoZIhvcNAQkIEw4xOTIuMTY4Ljg4LjI1NDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAtuJpgwpPUOxiYLxfMsL9kiQa5i0l2kmwYbyTm0XlqjEwOk8eRcyNQHybnyDBrmtuP4nlT/BkY3r0ZG7qD6fqg3xqZ5oOZe3w3vsvOF6U2G+eBbWAF8uu5BBT63r7rKcIg1mhEcAHCjEiGbDB4qQEKxEooELIlgjlEL3RAAdVkEECAwEAAaNTMFEwDwYDVR0TAQH/BAUwAwEB/zAfBgNVHSMEGDAWgBQcHfCtlFrqWC2SaBVzHS7XwxGF6DAdBgNVHQ4EFgQUHB3wrZRa6lgtkmgVcx0u18MRhegwDQYJKoZIhvcNAQEFBQADgYEAVP2ScAXMJllCPuvfyEyYVJW5H/iCgG1HV7CSI1IucGadRjJnd1hiIpoSXHRZ0wACkRXtsfJEnQmK52YHFUnlJr3nMlUluNZWroRMubAh3t04Rrr+OBIIc9q3BjTbzQtIE34u/o3hUya8mJKHf/+SxAhnbS/jOCkIT/PzwHwEmUw=-----END CERTIFICATE-----% RSA keypair 'ISR4321' is not exportable.

Copy and past into export.crt and inspect with openssl x509 -in export.crt -noout -text.

Import a TLS Certificate

You can also create a certificate on another machine and import into IOS.

Generate key pair:
```
openssl genpkey -algorithm rsa -pkeyopt rsa_keygen_bits:2048 \    -out key.pem  
```
By default, OpenSSL version 3 when invoked with openssl genpkey creates aPKCS#8 file (BEGIN PRIVATE KEY). (My version of) IOS only seems to acceptencrypted PKCS#1 (BEGIN RSA PRIVATE KEY).¹

Convert to PKCS#1 (-traditional) and encrypt with 3DES (-des3):

openssl pkey -in key.pem -traditional -des3 -passout pass:foobarbaz    -out key.enc.pem

Generate cert (valid for 10 years):

openssl req -x509 \    -key key.pem \    -out isr4300.crt \    -days 3700 \    -subj /CN=isr4300.test \    -addext 'subjectAltName=DNS:isr4300.test,IP:192.168.88.254'

Import this as a “trustpoint” into Cisco IOS:

ISR4300(config)# crypto pki import mytp pem terminal password foobarbaz

We see the following in running-config:

crypto pki trustpoint mytp enrollment pkcs12 revocation-check crl rsakeypair mytp!!crypto pki certificate chain mytp certificate ca 7FBC7EFBC685713BD7225B9AF3E8C7A3E0486078   3082032E 30820216 A0030201 0202147F BC7EFBC6 85713BD7 225B9AF3 E8C7A3E0   ...   quit

Enable TLS to Create Secure Server

ISR4321(config)# ip http secure-trustpoint mytpISR4321(config)# ip http secure-server

And disable cleartext HTTP:

ISR4321(config)# no ip http server

You should now be able to access WebUI over TLS:

$ curl -k https://192.168.88.254/webui

Caution: curl(1)’s -k, --insecure flag disables ALL TLS checks andjust accepts whatever certificate is presented to it: does not check whether thehostname matches nor whether it was signed by a trusted CA. Which is what youwant if you are using a self-signed certificate or a leaf certificate signed bya CA that is not in your certificate store, but otherwise do not use this.

You can see the difference in ASN.1 structure with openssl asn1parse -i -in <file>. (The -i flag “indents the output according to the ‘depth’ ofthe structures”.) ↩︎

Cisco IOS SSH

Mon, 10 Feb 2025 16:43:41 +0100

While connecting to a Cisco device out-of-band (i.e. not using IP) using aserial connection is the most secure way to configure the device, we also needto be able to connect over the network. Telnet works, but is not secure. So weneed SSH. This posts looks at how to set up SSH with username and password on aCisco device, how to use public-key certificates as a best practice toauthenticate and how to disable SSH altogether if it is not needed.

Cisco, of course, has pages upon pages of discussion on the most commonoperations with SSH with examples. They are the authoritative source, soconsult their documentation when necessary.

Set Up SSH with Username & Password

Before you can generate an RSA server host key, the hostname and the domain nameof the device need to be set. Together, those make a Fully-Qualified Domain Name(FQDN). That is how Cisco IOS identifies the key. Although it’s not entirelyclear to me why it needs this FQDN before generating an RSA key pair.

Router# conf tRouter(config)# hostname ISR4300ISR4321(config)# ip domain-name example.invalid

Now, we can generate the host key:

ISR4321(config)# crypto key generate rsaThe name for the keys will be: ISR4300.example.invalidChoose the size of the key modulus in the range of 360 to 4096 for yourGeneral Purpose Keys. Choosing a key modulus greater than 512 may takea few minutes.How many bits in the modulus [512]: 3072% Generating 3072 bit RSA keys, keys will be non-exportable...[OK]

Note: according to NIST, [3072 bit RSA keys are recommended and consideredsecure through 2030 and beyond][NIST]. The newest versions of IOS should alsosupport Ed25519.

SSH only works with usernames in addition to passwords¹:

ISR4321(config)# username stefan secret stefan

Enable only SSH authentication (not telnet or anything else):

ISR4321# conf tISR4321(config)# line vty 0 15ISR4321(config-line)# transport input sshISR4321(config-line)# login localISR4321(config-line)# no passwordISR4321(config-line)# end

If you set login with a password, 1 of 2 things might happen:

Configured users can use their username but share the same password
Nobody can log in, neither with their own password, nor with the sharedpassword.

So setting login local gets rid of this confusing mess: it will enableeverybody to use their own credentials.

For good measure, enable only SSH 2, and harden the configuration a bit:

ISR4321(config)# ip ssh version 2ISR4321(config)# ip ssh authentication-retries 2ISR4321(config)# ip ssh time-out 60

ip ssh authentication-retries 2 shuts down the connection after 2 failedauthentication attempts.

ip ssh time-out 60 shuts down the connection when not receiving authenticationfrom the client after a minute. (In my experimentation, it looks like IOS mightnot respect a timeout of less than 30 seconds, though.)

Show information about the configured SSH server:

ISR4321# sh ip sshSSH Enabled - version 2.0Authentication timeout: 60 secs; Authentication retries: 2

Show information about connected SSH clients:

SW01#sh sshConnection  Version  Mode  Encryption  Hmac       State             Username  0           1.99     IN    aes128-cbc  hmac-sha1  Session Started   stefan0           1.99     OUT   aes128-cbc  hmac-sha1  Session Started   stefan

Change how long an SSH or Telnet session lingers with exec-timeout <min> <sec>. By default, the SSH server disconnects after 5 minutes. Change it to 30minutes and 50 seconds:

ISR4321(config)# line vty 0 15ISR4321(config-line)# exec-timeout 30 50

Use Public Keys Instead of Passwords

To only accept public-key authentication, disabling password andkeyboard-interactive authentication:

ISR4321(config)# ip ssh server algorithm authentication publickey

Use an existing RSA key or generate a new one on your machine:

linux$ C="$(hostname)-$USER-remotehost-remoteuser-$(date -I)"linux$ ssh-keygen -t rsa -f ~/.ssh/"$C" -C "$C"

Now, import user’s public key into IOS. There is one thing to know, you cannotjust copy paste the entire RSA public key. IOS does not allow that amount ofinput. So you need to cut the public-key string up into bits of say 100characters each (apparently, up to 254 characters). Yeah, not very handy:

ISR4321(config)# ip ssh pubkey-chainISR4321(conf-ssh-pubkey)# username stefanISR4321(conf-ssh-pubkey-user)# key-stringISR4321(conf-ssh-pubkey-data)# AAAAB3NzaC1yc2EAAAADAQABAAABgQCq8kjk8UYdDYco+U9glX74dm0x1AzBJO0Gnj9tfjcR29HJC1TxauyDuni0QIfkoqr8AMUISR4321(conf-ssh-pubkey-data)# ua8zl5h8iTbQe1ErBPBLyevTEfCgA+JOMx9RaBytSg8S5JCR+ur65zuBLU3AhrogtadJiprr6SBcgfIh5Ryrj/oqRzBD+WTskQTdGNeBBn5ISR4321(conf-ssh-pubkey-data)# ... etc ...ISR4321(conf-ssh-pubkey-data)# exit

You could also specify the hash of your public key instead:

ISR4321(config)# ip ssh pubkey-chainISR4321(conf-ssh-pubkey)# username stefanISR4321(conf-ssh-pubkey-user)# key-hash ssh-rsa 7f7f72bfa276b87f44aaca363042c344 test

On my (old) version of IOS (16.06.04), the key-hash command takes a key-type(which can only be ssh-rsa) and the hex-encoded MD5 digest of your public key.Lastly, you can enter an optional comment (test in my case).

To calculate the hex-encoded MD5 digest of public key with openssl, first cutthe base64-encoded public key from the public-key file (2nd field):

$ cut -d ' ' -f2 < pub.key | \    openssl base64 -d | \    openssl dgst -md5MD5(stdin)= 7f7f72bfa276b87f44aaca363042c344

After pasting the base64-encoded public key or entering the MD5 hash, you cannow authenticate with your public key:

$ ssh -l stefan -i ~/.ssh/yourkey 192.168.88.254

Disabling SSH

I am a big fan of disabling services I do not use. That decreases attack surfaceand is good security hygiene. I cannot find a way to disable the listeningsocket on port 22, but I can disable any kind of meaningful interaction on theport:

ISR4321# conf tISR4321(config)# line vty 0 15ISR4321(config-line)# no transport input

IOS will now send a TCP FIN immediately when my SSH client starts to talk:

$ sudo tcpdump -nti eth1 port sshIP 192.168.88.252.38808 > 192.168.88.254.22: Flags [S], seq 1614430111, win 64240, options [mss 1460,sackOK,TS val 72081744 ecr 0,nop,wscale 7], length 0IP 192.168.88.254.22 > 192.168.88.252.38808: Flags [S.], seq 1208382352, ack 1614430112, win 4128, options [mss 1460], length 0IP 192.168.88.252.38808 > 192.168.88.254.22: Flags [.], ack 1, win 64240, length 0IP 192.168.88.252.38808 > 192.168.88.254.22: Flags [P.], seq 1:22, ack 1, win 64240, length 21: SSH: SSH-2.0-OpenSSH_9.9IP 192.168.88.254.22 > 192.168.88.252.38808: Flags [.], ack 22, win 4107, length 0IP 192.168.88.254.22 > 192.168.88.252.38808: Flags [FP.], seq 1, ack 22, win 4107, length 0IP 192.168.88.252.38808 > 192.168.88.254.22: Flags [F.], seq 22, ack 2, win 64239, length 0IP 192.168.88.254.22 > 192.168.88.252.38808: Flags [.], ack 23, win 4107, length 0

no transport input does disable telnet on port 23 entirely, though:

$ sudo tcpdump -nti eth1 port telnetIP 192.168.88.252.59455 > 192.168.88.254.23: Flags [S], seq 3152383467, win 1024, options [mss 1460], length 0IP 192.168.88.254.23 > 192.168.88.252.59455: Flags [R.], seq 0, ack 3152383468, win 0, length 0

Obtain Server’s Public Key

On the client side, we need to know the server’s public key, so we can be surenobody is playing miscreant in the middle:

ISR4321#sh ip sshSSH Enabled - version 1.99Authentication methods:publickey,keyboard-interactive,passwordAuthentication Publickey Algorithms:x509v3-ssh-rsa,ssh-rsaHostkey Algorithms:x509v3-ssh-rsa,ssh-rsaEncryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctrMAC Algorithms:hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96KEX Algorithms:diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1Authentication timeout: 120 secs; Authentication retries: 3Minimum expected Diffie Hellman key size : 2048 bitsIOS Keys in SECSH format(ssh-rsa, base64 encoded): TP-self-signed-3738675831ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn62IP88wHq9NilUvTuQ2Min+1CZQ+N8NtFyamsNaPgPibo71xq9joiCUQQcdi/tZYqsUfsGQQJ05QXlDj81z2EEnqZ+a3GK7XjulbStWdY2innc+UA//gYrhj21lar4PSsuFjz2VoBYxEoloh/Mk4o068IjeMNWrV3zADgAyHwigfYJyNdpoDkIM/PMb0J1XKR65/cCJz710olSBTorYdrmLEVp+mGnnGUfVQr4xovVeU/Sze1foutzVEkds6E9Olxn6bCqHjbiIetLtwiCYDEyLytjPA2SIYtmltCi/TByLl36cR6tNj6PiFMgvb1UkjwiF1Cse6WvKLzxS9k0M3

The line starting with “IOS Keys” is the server’s public key. If you copy it toa Unix box, you can calculate the SHA256 fingerprint as follows:

$ ssh-keygen -l -f isr4300.pub

Or using openssl:

$ openssl base64 -d < isr4300.pub | \    openssl dgst -sha256 -binary | \    openssl base64CRqX+T3a/48l6GT00Tgz3SmnkcNwlgt0B+vFI8VuzxI=

Either paste this hash when ssh(1) prompts to verify the remote host’sfingerprint (SHA256:CRqX+T3a/48l6GT00Tgz3SmnkcNwlgt0B+vFI8VuzxI=) or store thepublic key in ~/.ssh/known_hosts:

192.168.88.254 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn62IP88wHq9NilUvTuQ2Min+1CZQ+N8NtFyamsNaPgPibo71xq9joiCUQQcdi/tZYqsUfsGQQJ05QXlDj81z2EEnqZ+a3GK7XjulbStWdY2innc+UA//gYrhj21lar4PSsuFjz2VoBYxEoloh/Mk4o068IjeMNWrV3zADgAyHwigfYJyNdpoDkIM/PMb0J1XKR65/cCJz710olSBTorYdrmLEVp+mGnnGUfVQr4xovVeU/Sze1foutzVEkds6E9Olxn6bCqHjbiIetLtwiCYDEyLytjPA2SIYtmltCi/TByLl36cR6tNj6PiFMgvb1UkjwiF1Cse6WvKLzxS9k0M3

Now we can connect to our Cisco device with the peace of mind that we are onlytalking to the device, and nobody else.

Set privilege 15 to enter enable mode immediately after login. Setalgorithm-type scrypt to use a better PBKDF. ↩︎

How to Set Up Minicom for Serial Access

Mon, 10 Feb 2025 16:28:51 +0100

If you want to connect to a networking device from Cisco or MikroTik, you getout-of-band access to the device over a serial console port. minicom is one ofseveral programs that can be used to talk to these devices over a serialconnection.

[A serial console port] enables administration of a machine even if it has nokeyboard, mouse, monitor, or network attached to it.

Read Arch Linux wiki on working with the serial console.

Edit Configuration

To set up minicom enter:

$ minicom -s

minicom can only write its configuration file under /etc/minirc.dfl if youare root, though.

Under “Serial port setup”, set “Serial Device” to the /dev device file which isconnected to the router.

When I plug in my RJ-45-rollover-USB-converter cable, it shows up under/dev/ttyUSBn (where n is a number). When I connect with my USB mini-B cable,it shows up under /dev/ttyACMn. According to the Arch wiki, a traditionalhardware serial port, which, by default, modern computers do not ship withanymore, would show up under /dev/ttySn.

Make sure your user can access that port, though. On my Arch installation,/dev/ttyUSB0 is owned by root and part of group uucp. Add yourself to thegroup and restart your device or use sudo -g uucp minicom afterwards:

# usermod -aG uucp stefan

Set Bps/Par/Bits to 9600 8N1.

Increase Text Output Rate

To make text appear on the screen faster, increase the baud rate in IOS:

ISR4321# conf tISR4321(config)# line console 0ISR4321(config-line)# speed 115200

Then, press Ctrl+P in minicom and press E toset the speed to 115200.

Configure Static DNS Entries on Cisco IOS

Mon, 10 Feb 2025 16:21:42 +0100

You can configure Cisco’s IOS to remember IP addresses for host, without havingto resort to a DNS lookup. Very much indeed like adding an entry to your/etc/hosts file.

Router(config)# ip host example.invalid 192.168.1.1Router(config)# endRouter# sh hostDefault Domain is example.invalidName/address lookup uses domain serviceName servers are 255.255.255.255Codes: UN - unknown, EX - expired, OK - OK, ?? - revalidate       temp - temporary, perm - permanent       NA - Not Applicable None - Not definedHost             Port  Flags       Age  Type  Address(es)example.invalid  None  (perm, OK)  0    IP    192.168.1.1

Made a Typo? IOS Will Make You Wait for DNS Resolution

Mon, 10 Feb 2025 15:53:38 +0100

Another one of those Cisco IOS gotchas: when mistyping a command, you have towait a minute before you get back a prompt. How to get back your prompt rightaway?

To beginners with fat fingers like me, this happens all the time:

Router> fooTranslating "foo"...domain server (255.255.255.255)

IOS interprets this as a hostname to Telnet into. It will make you wait a minutebefore you can continue:

If the router has no DNS servers configured, another IOS default causes therouter to broadcast on the connected subnets looking for a DNS server toresolve names – with around a one-minute timeout waiting for a response.

Ctrl+C does nothing, but instead, we have to pressCtrl+^ to quickly cancel that command:

Router> fooTranslating "foo"...domain server (255.255.255.255) % Name lookup aborted

Type no ip domain-lookup in global configuration mode to disable all DNSlookups. I do not currently know if there are any downsides to this.

Improve Annoying Console Logging of Cisco IOS

Mon, 10 Feb 2025 15:41:55 +0100

Let’s have a quick look at how to keep Cisco IOS from interfering with yourprompt?

Improve Annoying Console Logging

I have set logging console to notifications or lower. Now, when exitingglobal configuration mode with ^Z or end and immediately starting to typeanother command such as sh run:

ISR4321(config-line)#end ISR4321#*Mar  9 09:27:18.751: %SYS-5-CONFIG_I: Configured from console by consolesh run

The logging interferes with what I am typing. Nothing too bad, I can type ^L toquickly get my prompt on a new line, but it would be convenient if IOS would dothat for me. Turns out, you can tell IOS to wait for the output of a showcommand to finish before printing anything on the console. This will also put ahalf-typed command on its own line, instead of appended to the console line.Much better.

ISR4321#conf tEnter configuration commands, one per line.  End with CNTL/Z.ISR4321(config)#line con 0ISR4321(config-line)#logging synchronousISR4321(config-line)#endISR4321#*Mar  9 09:29:08.489: %SYS-5-CONFIG_I: Configured from console by consoleISR4321#sh run

STP

Mon, 10 Feb 2025 10:42:13 +0100

In this post, we will have a look at the venerable Spanning Tree Protocol (STP),what it is used for, and how to configure and verify it on Cisco IOS andMikroTik.

STP Theory

STP (802.1D) prevents switching loops. Without it, a LAN with redundantlinks is susceptible to:

Broadcast storms, resulting in
MAC table instability; and
Multiple frame transmission

A broadcast storm is caused by a broadcast frame that is put on a LAN thathas a loop. It never dies, because it is flooded out every interface of aswitch, except the interface it came in on.

Because the same broadcast frame comes in on different ports of a switch withthe same source address, the switch keeps updating its MAC table: the entry isflapping.

Another consequence of a broadcast storm is that the same frame will bedelivered multiple times to the destination, and it might just be that theapplication cannot handle that correctly.

STP makes sure there is only path from a switch to a so-called root bridge. Theroot bridge gets elected among all available switches in the broadcast domain.

Root Bridge Election

The election works as follows: using a so-called Hello configuration BridgeData Protocol Units (BPDUs), each switch advertises itself as the rootbridge to its neighbors. It does this by including a root and sender Bridge ID(BID) that consists of a 2-byte priority field and a 6-byte system identifier:

A lower priority value indicates higher priority. The system identifier is equalto the switch’s burned-in MAC address.

The Hello BPDU includes the following fields:

Field	Description
Root BID	BID of switch the sender of this Hello BPDU believes to be the root bridge
Sender’s BID	BID of switch sending this Hello BPDU
Sender’s root cost	Root path cost between the sending switch and the current root
Root’s timer values	Hello timer, MaxAge timer, forward delay timer

A neighbor receives this BPDU and looks at the advertised root BID. If the rootBID is superior (meaning the priority is higher than what it currently thinksof as the root), it will:

change the sender BID in the Hello BPDU to its own BID
change the sender’s root cost to the cost between this switch and the root;and
forward the modified Hello BPDU out its other ports.

If the priority is lower (inferior), however, it will just ignore the BPDU.

By default, the 2-byte priority field in the BID is equal to 32,768 (or 0x8000in hex). In that case, the system ID (the burned-in MAC address) is used as atie breaker: the lower the 48-bit MAC address value, the higher its priority(1111.1111.1111 wins from 2222.2222.2222).

After a while, the root will be elected and the network will start toconverge (= stabilize on a network topology).

STP Roles and States

After the election, each switch’s interface takes on a role. These roles can be:

Root Port (RP): the single port on a switch facing the root switch thathas the least cost path to the root switch. Root ports are chosen by:
1. root path cost
2. BID of the upstream device
3. bridge port ID of the upstream device (consisting of a port priority and aunique ID, such as 128.1)
Designated Port (DP): a port (facing away from the root switch) on asegment that has the least cost path to the root switch. This port willforward the Hello BPDU onto the segment.

Because the elected root switch has the lowest cost path (0) to the root on allits ports, all of its ports will be designated ports (DPs).

The cost of a path to the root is determined by looking at the current speed(not maximum speed!) of the links that lie between a non-root switch’s port andthe root switch’s port. By default, traversing a 100 Mbps link carries a cost of19, while a 1000 Mbps costs 4:

Ethernet Speed	IEEE Cost 1998 (short)	IEEE Cost 2004 (long)
10 Mpbs	100	2,000,000
100 Mpbs	19	200,000
1,000 Mpbs/1 Gbps	4	20,000
10,000 Mpbs/10 Gbps	2	2,000
100,000 Mpbs/100 Gbps	N/A	200
1,000 Gbps/1 Tbps	N/A	20

On IOS, you can switch between the 2 short and long methods by:

Switch# spanning-tree pathcost method <short|long>

Set the interface cost for an interface:

Switch(config-if)# spanning-tree vlan 1 cost 123

Note that on IOS, you set the cost per VLAN on an interface. That is becauseIOS uses a modified version of STP called the Per-VLAN Spanning Tree Protocol(PVST+)¹, which is a topic for a separate blog post.

With STP, every interface can be in state:

Blocking (stable)
Listening (transient)
Learning (transient)
Forwarding (stable)
Disabled (stable)

(Between parentheses whether the state is a stable state or an intermediatestate towards a stable state.)

Root Ports and Designated Ports will be forwarding frames. If a port is not aRP or a DP, it will be blocked. That means that it will not receive nor transmituser frames. It will still receive BPDUs, though.

If an interface is administratively shut down, its state will be “disabled”.

On IOS, you can check on all of this with show spanning-tree:

Switch#sh spanning-tree VLAN0001  Spanning tree enabled protocol rstp  Root ID    Priority    24577             Address     000B.BE6A.4623             Cost        19             Port        1(FastEthernet0/1)             Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec  Bridge ID  Priority    28673  (priority 28672 sys-id-ext 1)             Address     00D0.BA39.4B87             Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec             Aging Time  20Interface        Role Sts Cost      Prio.Nbr Type---------------- ---- --- --------- -------- ---------------------------Fa0/2            Desg FWD 19        128.2    P2pFa0/1            Root FWD 19        128.1    P2p

Timers

A root bridge sends a Hello BPDU every 2 seconds by default. This Hello Time iscarried in the BPDU. It also contains a value for MaxAge, which defines how longswitches should wait after ceasing to hear Hellos before trying to change theSTP topology. By default, it is 10 times the Hello Time. So a switch will wait20 seconds for Hellos to come in before taking action.

The forward delay timer is used to determine how long a switch will stay inlistening and learning states after deciding to transition a port from blockingto listening.

In the listening state, the switch processes BPDUs but does not forward any userframes. It tries to determine whether the interface should indeed be forwardingor should return to the blocking state. It makes this decision based on thereceived BPDUs. Also, the switch removes unused MAC table entries for which noframes are received during this period.² Inthe learning state, the switch begins to learn the MAC addresses of framesreceived on the interface.

All in all, a switch could take 50 seconds (20 MaxAge, 2 times the forward delayfor listening and learning) before an interface transitions from blocking toforwarding.

Edge Devices

Only switches participate in STP. Edge devices (PCs, laptops, IoT, etc.) do not.But the links that connect edge devices to switches will, by default, still gothrough the motions by first entering listening state and then learning state.You can short-circuit this process by configuring spanning-tree portfast onthe interface. The link will now transition to forwarding immediately.

In the event you do connect a switch to a port that was configured withportfast, you can inadvertently create a loop. In that case, you should alsoenable BPDU Guard on the port by setting spanning-tree bpduguard enable. Ifthe port detects a BPDU coming in, it will automatically disable itself.

MikroTik

On MikroTik’s RouterOS 7, make sure you have STP enabled:

[user@rb] /interface/bridge/set BR1 protocol-mode=rstp

(Even though this post is about the traditional IEEE 802.1D STP,please enable the more modern (faster!) RSTP instead.)

Check the status of a bridge:

[u@rb] /interface/bridge/monitor BR1                  state: enabled                     current-mac-address: 48:A9:8A:11:11:11            root-bridge: yes                              root-bridge-id: 0x8000.48:A9:8A:11:11:11         root-path-cost: 0                                     root-port: none                                 port-count: 23                        designated-port-count: 5                                  fast-forward: no

It indicates this bridge is the root bridge (current-mac-address == root-bridge-id) and has a BID of 0x8000.48:A9:8A:11:11:11: here, 0x8000 isthe priority in hex (= 32,768 in decimal; change with set BR1 priority=0xNNNNto a multiple of 4,096) and the system ID is the 48:A9:8A:11:11:11.

Check an individual port:

[u@rb] /interface/bridge/port> monitor [find interface=ether17]            interface: ether17               status: in-bridge                port-number: 16                              role: designated-port            edge-port: yes              edge-port-discovery: yes              point-to-point-port: yes                     external-fdb: no                      sending-rstp: yes                         learning: yes                       forwarding: yes                 actual-path-cost: 10                  hw-offload-group: switch1

It automatically discovered that interface ether17 is connected to an edgedevice. If you want to be specific, you can set PortFast by set [find interface=ether17] edge=yes. Set BPDU Guard with set [find interface=ether1] bpdu-guard=yes.

Change between the long and short automatic costs based on interfaces speeds:

[user@rb] /interface/bridge/set BR1 port-cost-mode=long

And a bit of warning: RouterOS does not currently play well with Cisco’sPVST. It will treat it as regular multicast traffic.

The + at the end indicates that IEEE 802.1Q is used fortrunking instead of Cisco’s older, proprietary ISL protocol. ↩︎
This state was subject to dispute, asit is unclear whether it matters if STP is learning MAC addresses or nothere. It is telling that RSTP has done away with this state. ↩︎

Check OCSP With OpenSSL CLI Tool

Sat, 08 Feb 2025 12:04:49 +0100

How to query an OCSP responder to check the validity of a certificate?

Get the cert chain:

$ H=github.com$ echo Q | \    openssl s_client -showcerts -servername $H -host $H -port 443 | \    awk 'BEGIN{n=0}; /-BEGIN/,/-END/{ print >> n ".pem" }; /-END/{n++}'

-showcerts:
Displays the server certificate list as sent by the server: it only consistsof certificates the server has sent (in the order the server has sent them).It is not a verified chain.
I use awk(1) to filter the PEM certificates from the other output. It sets acounter to 0. It then appends every line between -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- to a file. It increments thenumber after every -----END CERTIFICATE-----.
This will create as many files as the certificate chain is long. The file0.pem contains the “certificate at depth 0”: the leaf certificate. The file1.pem contains the “certificate at depth 1”: the issuer of the leafcertificate. Caveat: this may not be true, as the order depends on theserver. Check the subject and issuer of each certificate and rename the filesaccordingly:
```
$ for f in *.pem; do       echo "$f"      openssl x509 -in "$f" -noout -subject -issuer      echo  done
```
When openssl s_client receives Q, it tears down the TLS connection to theremote server.

Get the OCSP responder URI from the leaf cert:

$ openssl x509 -in 0.pem -noout -ocsp_urihttp://ocsp.sectigo.com

Let’s store it in a variable for easier access:

$ ocsp_uri=$(openssl x509 -in 0.pem -noout -ocsp_uri)

We now have everything we need:

$ openssl ocsp -issuer 1.pem -cert 0.pem -url $ocsp_uri -textOCSP Request Data:    Version: 1 (0x0)    Requestor List:        Certificate ID:          Hash Algorithm: sha1          Issuer Name Hash: CF94DC5C304AA79485721F956E67895AC21657DD          Issuer Key Hash: F6850A3B1186E1047D0EAA0B2CD2EECC647B7BAE          Serial Number: AB6686B5627BE80596821330128649F5    Request Extensions:        OCSP Nonce:            041038CA4568E928AE97DD06259DBF6CBE0AOCSP Response Data:    OCSP Response Status: successful (0x0)    Response Type: Basic OCSP Response    Version: 1 (0x0)    Responder Id: F6850A3B1186E1047D0EAA0B2CD2EECC647B7BAE    Produced At: Feb  7 06:04:44 2025 GMT    Responses:    Certificate ID:      Hash Algorithm: sha1      Issuer Name Hash: CF94DC5C304AA79485721F956E67895AC21657DD      Issuer Key Hash: F6850A3B1186E1047D0EAA0B2CD2EECC647B7BAE      Serial Number: AB6686B5627BE80596821330128649F5    Cert Status: good    This Update: Feb  7 06:04:44 2025 GMT    Next Update: Feb 14 06:04:43 2025 GMT    Signature Algorithm: ecdsa-with-SHA256    Signature Value:        30:44:02:20:08:b0:f7:00:53:f9:d4:6c:7f:a0:91:ac:0a:91:        af:a1:34:e2:c5:2c:dc:14:a0:1c:de:e9:be:22:6f:25:d8:2f:        02:20:0d:09:7e:57:43:59:ce:c4:65:f1:4b:f9:3d:f4:46:82:        78:01:4c:53:ca:2f:45:b3:35:0c:83:54:53:f0:68:edWARNING: no nonce in responseResponse verify OK0.pem: good        This Update: Feb  7 06:04:44 2025 GMT        Next Update: Feb 14 06:04:43 2025 GMT

In the request and response, the “Hash Algorithm” indicates SHA-1 was used togenerate hashes.

The serial number is the serial number of the leaf cert:

$ openssl x509 -in 0.pem -noout -serialserial=AB6686B5627BE80596821330128649F5

The “Issuer Name Hash” and “Issuer Key Hash”:
issuerNameHash is the hash of the issuer’s distinguished name(DN). The hash shall be calculated over the DER encoding of theissuer’s name field in the certificate being checked.
issuerKeyHash is the hash of the issuer’s public key. The hashshall be calculated over the value (excluding tag and length) ofthe subject public key field in the issuer’s certificate.
–RFC 6960
This is hard to do by hand, so we use openssl(1) once more:
```
$ openssl x509 -in 1.pem -noout -ocspidSubject OCSP hash: CF94DC5C304AA79485721F956E67895AC21657DDPublic key OCSP hash: F6850A3B1186E1047D0EAA0B2CD2EECC647B7BAE
```
Note that these values correspond to what is in our OCSP request and response.
(I did not get output from LibreSSL 4.0.0 with -ocspid, even though theoption is mentioned in its manpage.)
“WARNING: no nonce in response” means that the server did not include ournonce in their response. Nonces are used to prevent replay attacks. Theyincrease the load on the OCSP responder, however. That is why responders maychoose to ignore the nonce and return a canned response.

Rapid Spanning Tree Protocol (RSTP)

Sun, 02 Feb 2025 16:36:16 +0100

I previously discussed the plain-old Spanning Tree Protocol (STP). In thispost, I will look at the differences introduces by its successor, the RapidSpanning Tree Protocol (RSTP).

The Case for RSTP

STP required 50 seconds to get a port in forwarding state: it waits 20 secondsfor the MaxAge (by default 10 times the value of the Hello Timer) timer toexpire. At that point, it knows it has to change the topology. It will spend theForward Delay in a listening state (15 seconds by default) and then anotherForward Delay in learning mode.

All that waiting is tedious: 50 seconds is a long time to be without networkconnectivity. Devices that live off of edge ports expect to be able to do DHCPand are unable to. Apparently, some DHCP client implementations just give up ifthey cannot Discover a DHCP server within those 50 seconds.

So RSTP promises faster convergence. It does this by introducing new roles. STPhad Root Ports (ports facing toward the root bridge that had the least pathcost from that port to the root) and Designated Ports (ports facing away fromthe root bridge that had the least path cost to the root bridge on that segmentand that was therefore designated to put the BPDUs on the segment).

RSTP introduces the Alternate and Backup Port roles. Alternate Portsfunction as secondaries to the Root Ports (on a point-to-point link type).They have the least cost path to the root after the Root Port. Backup Ports areonly relevant to shared links. This means that when you have multiple links intothe same segment (the same collision domain), all but one of those links will beblocking (or discarding in RSTP parlance). One of the blocked ports will be aBackup Port. Shared access to a collision domain only happens when usinghubs¹ (but see below for an interesting inter-op case with MikroTik bridges).

So: Alternate Ports step in for failing Root Ports, Backup Ports step in forfailing Designated Ports.

The idea of assigning Alternate Ports is that when a Root Port goes down, theAlternate Port can transition to the forwarding state without any delays. Samewith Backup Ports.

STP had 5 states a port could be in:

Blocking
Listening
Learning
Forwarding
Disabled

RSTP simplifies this to 3:

Discarding
Learning
Forwarding

The discarding state now encapsulates the blocking and disabled states. Thelistening state is simply not used anymore.

Key Differences between STP and RSTP

RSTP detects root failure in 3 Hello times (by default: 3 * 2 = 6 seconds),instead of ten Hello times (by default: 10 * 2 = 20 seconds).
RSTP may send BPDUs in the direction of the root bridge, confirming to theupstream bridge that it has received superior information. The bridgereceiving this confirmation can rapidly transition the port to the forwardingstate, skipping the listening/learning states.

And, as everybody keeps saying in instruction videos and podcasts, theseare exceedingly rare these days. I’ve never even seen a hub in my lifeexcept on network diagrams. ↩︎

IEEE RSTP and Cisco RPVST+ Interop

Sat, 01 Feb 2025 11:35:18 +0200

Let’s have a look at what happens when we plug some MikroTik devices into aCisco switch and string them together in a loop topology. What happens when RSTPand RPVST+ need to work together? Do we get a broadcast storm? Why (not)?

Interaction of RSTP with RPVST+

RSTP is VLAN-agnostic. The 12-bit System ID Extension in the BID is set to allzeroes 0000.0000.0000. Cisco’s proprietary (R)PVST+, on the other hand,populates this field with the ID of the VLAN on which the BPDU containing theBID is broadcast.¹

RSTP sends untagged multicasts to 0180.C200.0000 onto the native VLAN,creating a single Common Spanning Tree (CST). Cisco’s RPVST+ sends taggedmulticast frames to 0100.0CCC.CCCD on every VLAN, creating a separate tree forevery VLAN. So, if you have 10 VLANs defined, RPVST+ will send 10 BPDU Hellosevery Hello interval.

This also means that Cisco and MikroTik gear do not play (well) together.MikroTik treats Cisco’s proprietary (R)PVST+ frames as it would any othermulticast and forwards it out all ports except the port it received theframes on. Cisco will discard multicast traffic sent to the IEEE multicastaddress if VLAN 1 is not the native VLAN and not allowed on the trunk.

I found the following image on the Cisco Community: it helps explain thesituation when the native VLAN is the default 1 and when it is something else:

This is the topology we will use to think about different scenario’s:

I won’t repeat the interface names (ether5) in the scenario topologies,because for MikroTiks, they are pretty much meaningless. Plus the maximum speedsof the interfaces can be deduced from looking at the path cost. All paths costsare in the traditional IEEE 1998 form (see this post) where 100 Mbps linkshave a path cost of 19 and 1 Gbps links a cost of 4.

The MikroTiks, a hEX and hAP lite, run 7.19.1 firmware. The Cisco, a3650C, runs a venerable 15.0(2)SE2.

Scenario 1: Everybody Uses Native VLAN 1

In this scenario:

VLAN 1 is the native VLAN
VLAN 20 is tagged

What happens in VLAN 1:

Cisco sends untagged VLAN 1 BPDU both to IEEE and Cisco addresses
Cisco processes untagged RSTP BPDUs coming from MikroTiks under VLAN 1
STP converges and blocks a port to prevent a switching loop

What happens in VLAN 20:

Cisco starts out assuming it is the root bridge on VLAN 20
Cisco sends tagged VLAN 20 BPDUs² to the Cisco address
MikroTiks see Cisco address as just another multicast address and flood it
For a brief period of time, before VLAN 1 STP converges and blocks a port,Cisco receives its own BPDU on its other interface and decides that it’sconnected to a shared segment
Cisco transitions the port with the lowest port priority to the backup roleand discards incoming traffic on that port
After VLAN 1 converges and a port is blocked, Cisco stops receiving its ownBPDUs on the backup port, decides that a topology change has taken place andtransitions the port’s role from backup to designated port

In the (R)PVST+ world, the Cisco switch will be the root on all VLANs becausethere are no other contenders: the switch will never hear anything from anybody,because the MikroTik at the bottom has blocked its outgoing interface to theCisco switch.

Switch#show spanning-tree vlan 20VLAN0020  Spanning tree enabled protocol rstp  Root ID    Priority    32778             Address     1ce6.c707.ff00             This bridge is the root             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec  Bridge ID  Priority    32788  (priority 32768 sys-id-ext 20)             Address     1ce6.c707.ff00             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec             Aging Time  300 secInterface           Role Sts Cost      Prio.Nbr Type------------------- ---- --- --------- -------- --------------------------------Gi0/1               Desg FWD 4         128.9    P2pGi0/2               Back BLK 19        128.10   P2p

Interface           Role Sts Cost      Prio.Nbr Type------------------- ---- --- --------- -------- --------------------------------Gi0/1               Desg FWD 4         128.9    P2pGi0/2               Desg FWD 19        128.10   P2p

Any loops?

After convergence in VLAN 1, a port is blocked and no loop is possible
This blocked port also prevents loops on RPVST+ VLAN 20, even though the Ciscoput both its ports in forwarding state

Scenario 2: Native VLAN 20, VLAN 1 Allowed on Cisco Trunk

Things get more interesting when you change the native VLAN.

In this scenario:

VLAN 20 is the native untagged VLAN
VLAN 1 is just another VLAN allowed on the trunk

With regard to VLAN 1:

Cisco sends untagged VLAN 1 BPDUs to the IEEE address
Cisco sends tagged VLAN 1 BPDUs to the Cisco address
MikroTiks process untagged VLAN 1 BPDUs as RSTP messages
MikroTiks flood tagged VLAN 1 BPDUs just like any other multicast traffic
The network converges and the same port is blocked as in scenario 1

With regard to VLAN 20:

Cisco sends untagged VLAN 20 BPDUs to the Cisco address
MikroTiks flood tagged VLAN 1 BPDUs just like any other multicast traffic
Because a port gets blocked by RSTP on the MikroTiks, the Cisco port with thelowest priority first becomes a backup port and soon afterwards a designatedport, just as in scenario 1

Again, loops are prevented: a flooded broadcast, unknown unicast or multicast(BUM) frame arriving on the MikroTiks will not result in a broadcaststorm. Also, any BUM traffic arriving on the Cisco switch from any of theother VLANs via access links will also not cause a broadcast storm because theMikroTik blocks all traffic on all VLANs.

Scenario 3: Native VLAN 20, VLAN 1 Disallowed on Cisco Trunk

In this scenario:

VLAN 20 is the native untagged VLAN
VLAN 1 is not allowed

With regard to VLAN 1:

Cisco does not initiate any BPDUs to the IEEE address
Cisco drops received BPDUs addressed to the IEEE address
MikroTiks see that Cisco does not participate in the root-bridge election byinitiating its own BPDUs or forwarding them, so the MikroTiks ports facing theCisco switch automatically become the designated ports

With regard to VLAN 20:

Cisco sends untagged VLAN 20 BPDUs to the Cisco address
MikroTiks flood untagged VLAN 20 BPDUs just like any other multicast traffic
Cisco sets a port with the lowest priority to be the backup port and discardstraffic

A switching loop is prevented, because all traffic (both tagged and untagged)coming in on the Cisco switch will not be switched out of the (discarding)backup port.

In the image above, note that it merges 2 views. Cisco views the world as theroot bridge of the native VLAN 20 with no other contenders. Because it receivesits own RPVST+ BPDU on its other port, it designates the other port as a backupand blocks traffic over it. The MikroTiks have another view of the native VLAN(might be or not VLAN 20): because the Cisco does not initiate or forward IEEERSTP BPDUs, it considers ports going out to the Cisco to be designated ports andforward traffic over them. So we end up with 2 isolated root bridges. But all iswell and no loops occur.

Conclusion

So is it safe to mix MikroTiks and Cisco bridges with regard to switching loops?The answer seems to be yes, at least for the simple 3-bridge topology anddifferent configurations I examined here. As to more complex topologies andconfiguration, we can say a couple of things:

Cisco’s proprietary (R)PVST+ BPDUs will always be flooded by MikroTik bridges
Those MikroTik bridges build their own CST, making sure there is no loopbetween them, blocking ports where necessary
If the Cisco bridges do not forward the IEEE RSTP BPDUs (when they do notallow VLAN 1), the MikroTik bridges will forward on those interfaces.
That would be problematic, were it not that the Cisco switches consider thoseports to be connected to a shared segment (a hub, basically), transition themto backup roles and start discarding traffic.

It’s another question entirely whether all of this results in the most efficienttraffic flows. If analysis shows it isn’t, it would be best to consider theIEEE’s standardized Multiple STP on both the MikroTiks and Cisco’s. Butthat’s a topic for another time.

Unless that VLAN ID happens to be the native VLAN ID, of course. ↩︎
(R)PVST+ indicates the originating VLAN ID in its BPDUs by using aso-called TLV (type-length-value) extension header. ↩︎

Open a ZIP File With Any Extension In Vim

Sat, 01 Feb 2025 09:05:39 +0100

What to do when you use Vim to open a zip file, but the file extensionis not .zip or one of the other extensions known to Vim’s zip plugin?

Displaying ZIP File Content With Vim

Out of the box, Vim is able to open zip files and display its contents.You can even edit zipped files and save them.

Vim recognizes the following file extensions as containing zippedcontent:

:echo g:zipPlugin_ext*.aar,*.apk,*.celzip,*.crtx,*.docm,*.docx,*.dotm,*.dotx,*.ear,*.epub,*.gcsx,*.glox,*.gqsx,*.ja,*.jar,*.kmz,*.odb,*.odc,*.odf,*.odg,*.odi,*.odm,*.odp,*.ods,*.odt,*.otc,*.otf,*.otg,*.oth,*.oti,*.otp,*.ots,*.ott,*.oxt,*.potm,*.potx,*.ppam,*.ppsm,*.ppsx,*.pptm,*.pptx,*.sldx,*.thmx,*.vdw,*.war,*.wsz,*.xap,*.xlam,*.xlsb,*.xlsm,*.xlsx,*.xltm,*.xltx,*.xpi,*.zip

However, when we have a zip file that does not end in one of theseextensions, we need to instruct Vim to execute the zip pluginregardless.

What You’ll See When The File Extension Is Not Supported

Say, we have a zip file called archive:

$ file --mime archive archive: application/zip; charset=binary

Opening it with Vim results in:

PK^C^D^@^@^@^@^@^LKAZ^@^@^@^@^@^@^@^@^@^@^@^@^E^@^\^@a.txtUT^@^C7Ú<9d>g7Ú<9d>gux^K^@^A^Dè^C^@^@^Dè^C^@^@PK^C^D... snip ...

Not what we want.

Getting Vim to Execute the ZIP Plugin Regardless of Extension

Let’s replace the buffer with zip plugin’s file browser:

:enew!:call zip#Browse("/path/to/archive")

Let’s break these commands down:

:enew! creates a new buffer, hiding the previous buffer. Theexclamation mark discards any changes made to the original buffer.
:call calls a function. The zip#Browse() function takes apath to a zip file and shows the content of the unzipped file.

Now we get the desired result:

" zip.vim version v34" Browsing zipfile archive" Select a file with cursor and press ENTERa.txtb.txt

Always Opening Your File Extension With the ZIP Plugin

If you always want to open your file extension to be opened with the zipplugin, you can add the extension to this list and put the following inyour .vimrc:

let g:zipPlugin_ext ..= ",*.mmm"

..= is the string concatenation operator. See :h expr-...

Build Kotlin JMH Benchmarks With Maven

Fri, 24 Jan 2025 14:52:10 +0100

In this post, we will look at how to set up the Maven infrastructure so we canbenchmark Kotlin code with JMH (the Java Microbenchmarking Harness).

This post does not look in-depth on how to write JMH benchmarks, but rather howto build and run them.

Create a Sample JMH Project

Helpfully, the JMH project has a Maven archetype for Kotlin:

$ mvn archetype:generate \    -DinteractiveMode=false \    -DarchetypeGroupId=org.openjdk.jmh \    -DarchetypeArtifactId=jmh-kotlin-benchmark-archetype \    -DgroupId=com.relentlesscoding \    -DartifactId=benchmarks \    -Dversion=1.0

This creates a sample project where we can play around with a simple benchmarkwritten in Kotlin.

Things to Keep in Mind When Creating JMH Benchmarks

Benchmarks look like this:

package com.relentlesscodingimport org.openjdk.jmh.annotations.Benchmarkopen class MyBenchmark {    @Benchmark    fun benchmarkMethod() {    }}

See a more elaborate benchmarkexamplethat benchmarks signing efficiency of different-size RSA keys at the end of thispost. Also see the examples the JMH team provides.

Notice that the class needs to be declared open:

Benchmark classes should not be final.                                                            [com.sample.RSASigningMessageBenchmark]

This is because JMH needs to be able to subclass it to add instrumentation codeand implement the benchmarking machinery.

If you or your IDE objects to opening classes, you could also use the all-opencompiler plugin that is part of the kotlin-maven-plugin:

<plugin>  <groupId>org.jetbrains.kotlin</groupId>  <artifactId>kotlin-maven-plugin</artifactId>  <!-- ... snip ... -->  <configuration>    <compilerPlugins combine.children="append">      <plugin>all-open</plugin>    </compilerPlugins>    <pluginOptions>      <option>        all-open:annotation=org.openjdk.jmh.annotations.BenchmarkMode      </option>    </pluginOptions>  </configuration>  <dependencies>    <dependency>      <groupId>org.jetbrains.kotlin</groupId>      <artifactId>kotlin-maven-allopen</artifactId>      <version>${kotlin.version}</version>    </dependency>  </dependencies></plugin>

Building Kotlin JMH Benchmarks with Maven

Compiling the Kotlin benchmark requires the following steps:

Compile the Kotlin source to bytecode:

<plugin>  <artifactId>kotlin-maven-plugin</artifactId>  <groupId>org.jetbrains.kotlin</groupId>  <version>${kotlin.version}</version>  <executions>    <execution>      <id>process-sources</id>      <phase>generate-sources</phase>      <goals>        <goal>compile</goal>      </goals>      <configuration>        <sourceDirs>         <sourceDir>${project.basedir}/src/main/kotlin</sourceDir>        </sourceDirs>      </configuration>    </execution>  </executions></plugin>

Run the JMH bytecode generator, which:

Analyzes the compiled (Kotlin) benchmark classes
Generates additional Java source files for benchmark infrastructure
Creates metadata for benchmark registration and execution

<plugin>  <groupId>org.codehaus.mojo</groupId>  <artifactId>exec-maven-plugin</artifactId>  <executions>    <execution>      <id>generate-benchmark-code</id>      <phase>process-sources</phase>      <goals>        <goal>java</goal>      </goals>      <configuration>        <includePluginDependencies>true</includePluginDependencies>        <mainClass>          org.openjdk.jmh.generators.bytecode.JmhBytecodeGenerator        </mainClass>        <arguments>          <!-- compiled-bytecode-dir -->          <argument>${project.basedir}/target/classes/</argument>          <!-- output-source-dir -->          <argument>              ${project.basedir}/target/generated-sources/jmh/          </argument>          <!-- output-resource-dir -->          <argument>${project.basedir}/target/classes/</argument>          <!-- generator-type -->          <argument>default</argument>        </arguments>      </configuration>    </execution>  </executions>  <dependencies>      <dependency>          <groupId>org.openjdk.jmh</groupId>          <artifactId>jmh-generator-bytecode</artifactId>          <version>${jmh.version}</version>      </dependency>  </dependencies></plugin>

Add the generated JMH Java files as a source to the Maven reactor (otherwiseMaven ignores the source):

<plugin>  <groupId>org.codehaus.mojo</groupId>  <artifactId>build-helper-maven-plugin</artifactId>  <version>1.8</version>  <executions>    <execution>      <id>add-source</id>      <phase>process-sources</phase>      <goals>        <goal>add-source</goal>      </goals>      <configuration>        <sources>          <source>            ${project.basedir}/target/generated-sources/jmh          </source>        </sources>      </configuration>    </execution>  </executions></plugin>

Have the maven-compiler-plugin compile the whole bunch:

<plugin>  <groupId>org.apache.maven.plugins</groupId>  <artifactId>maven-compiler-plugin</artifactId>  <version>3.8.0</version>  <configuration>    <compilerVersion>${javac.target}</compilerVersion>    <source>${javac.target}</source>    <target>${javac.target}</target>    <compilerArgument>-proc:none</compilerArgument>  </configuration></plugin>

Finally, you can use the maven-assembly-plugin or the maven-shade-pluginto create an uberjar (jar with dependencies). I prefer the assembly pluginbecause it is simpler (the JMH archetype provides an example with themaven-shade-plugin, though, if you are interested):

<plugin>  <groupId>org.apache.maven.plugins</groupId>  <artifactId>maven-assembly-plugin</artifactId>  <executions>    <execution>      <id>make-uberjar</id>      <phase>package</phase>      <goals>        <goal>single</goal>      </goals>      <configuration>        <archive>          <manifest>            <mainClass>org.openjdk.jmh.Main</mainClass>          </manifest>        </archive>        <descriptorRefs>          <descriptorRef>jar-with-dependencies</descriptorRef>        </descriptorRefs>      </configuration>    </execution>  </executions></plugin>

Make sure to put org.openjdk.jmh.Main as the Main-Class in the jarmanifest, and not your own main class. This will allow you to tweak the JMHparameters. See java -jar benchmarks-jar-with-dependencies -h.

You are now ready to run the jar with java -jar benchmarks-jar-with-dependencies.jar from the target/ directory.

Using a Annotation Processor Instead

This approach above is pretty explicit. Alternatively, we could use thejmh-generator-annprocess annotation processor with deprecated kapt orthe Kotlin Symbol Processing (KSP) API (which I have yet to use). Thatworks pretty well for a Kotlin-only project, but is more difficult to getworking with mixed sources. Also, it might complicate debugging when things gowrong. The benefit is that, this way, we could skip 2 steps in the Maven build:the explicit generation of the bytecode in the exec-maven-plugin byJmhBytecodeGenerator and adding the sources to the build in thebuild-helper-maven-plugin.

The following would enable kapt execution as part of thekotlin-maven-plugin:

<plugin>  <groupId>org.jetbrains.kotlin</groupId>  <artifactId>kotlin-maven-plugin</artifactId>  <executions>    <!-- ... snipped compilation execution ... -->    <execution>      <id>kapt</id>      <goals>          <goal>kapt</goal>      </goals>      <configuration>        <sourceDirs>          <sourceDir>src/main/kotlin</sourceDir>        </sourceDirs>        <annotationProcessorPaths>          <annotationProcessorPath>            <groupId>org.openjdk.jmh</groupId>            <artifactId>jmh-generator-annprocess</artifactId>            <version>${jmh.version}</version>          </annotationProcessorPath>        </annotationProcessorPaths>      </configuration>    </execution>  </executions>  <dependencies>    <dependency>      <groupId>org.openjdk.jmh</groupId>      <artifactId>jmh-generator-annprocess</artifactId>      <version>${jmh.version}</version>    </dependency>  </dependencies></plugin>

Example: Benchmark Creating Signatures With Different-Length RSA Keys

package com.relentlesscoding.benchmarksimport org.openjdk.jmh.annotations.*import org.openjdk.jmh.infra.Blackholeimport java.util.*import java.util.concurrent.TimeUnitimport java.security.*import java.security.spec.PKCS8EncodedKeySpecimport java.security.interfaces.RSAPrivateCrtKey// override with -bm <thrpt | avgt | sample | ss | all>@BenchmarkMode(Mode.Throughput)// override with -wi <int>@Warmup(iterations = 1)  // override with -i <int>@Measurement(iterations = 1)  // override with -tu <s | ms | us | ns>@OutputTimeUnit(TimeUnit.MILLISECONDS)  @Fork(1)open class RSASigningMessageBenchmark {  @State(Scope.Benchmark)  open class BenchmarkState {    private val aPKCS8Encoded512BitRSAPrivKey = "MIIBVgIBADANBgkqhkiG9w0BAQEFAASCAUAwggE8AgEAAkEA0fKfr7qTLwt+UBk8oTVks1MmO6L6Fm/15xG8jLx/y274swbQUXO0RZtZWMr8+W00FgdegRNMNgxdIMfZFyfFzQIDAQABAkEAnjos/1Ot+Za/674ZY6XJ7xyLhAagVKisuyky4R5vcfEjK4GgpQO8oOTVP09w8SOZdSa5Vb+6coFpvWs5zwHaAQIhAOhDNww0vkua0HMw/8+jEN9ZpY5NDGCD2mIkNGcaKg75AiEA52eTqbTcC4O5yLHDIblvw6YSylVmdATQOi8i9PZL3nUCIQC70icAyuIb94ybqkMjoMUzKKZ1pZ7dqaJ+/LIXshPS2QIgGNCusSBICKQTpEYL2u374ktI8JG/7uklO1gas5JGCJECIQDEAyVyFTisbbFHqQCUdduakO2XD1yZ5a4ACSTdJoxkBg=="    private val aPKCS8Encoded768BitRSAPrivKey = "MIIB5wIBADANBgkqhkiG9w0BAQEFAASCAdEwggHNAgEAAmEAy5XJREg4ki+HZAzgTF7r6wDUy0gi1Njt4svAWfWT3r/vNGcVep3aGEfP5C262257soZMw9wmpbCtFU97FH8nJ3tb5yc6qzvyVnk56M3WMStLtC5dPenExbDLst08co/fAgMBAAECYQCDtXkLiunGcadW7BmkbviUBeqlRRr7twhX5Nehm4Y54tR/g31a4Yq6kKMHjSpJUjTeQ5EmmqxVVErwxjgJYj3FR+aoy8hk39tTW0OytgXGAmYUyRqpQq7/o63H+zvMMPECMQD4K31WQLRfNsyCDsPaHwzL09H6jqtObYXUCDdNeff+9dE8611qGEA94ypOPQpbmrcCMQDSAi2A2E7+9zjTDdkGQTfhTguyuesLMmXZNU6sDeFGdwZpI1P0/pbBw0XfendfLBkCMQCofWZgPBf6GQtqNboVCkW20T5b3adC3SsiVN2vNWMBcEW6FZZbpNFg8y1S5zB0FysCMQCbT+j/JPonLgbkb5VVPt5oziNwpnbh7P/Nx9LLA+jbCCPBldL9mVs9KYF/aT7nL+ECMQDeYptxqot70d4nWsJoZ5QE8RiU6y7ZGjbDh48QQ87HXjj0ezvr/qMg+eKr8U9/mWs="    private fun parseKey(pkcs8EncodedRSAPrivateKey: String): RSAPrivateCrtKey {      val decoded = Base64.getDecoder().decode(pkcs8EncodedRSAPrivateKey)      val keySpec = PKCS8EncodedKeySpec(decoded)      val keyFactory = KeyFactory.getInstance("RSA")      return keyFactory.generatePrivate(keySpec) as RSAPrivateCrtKey    }    val a512BitRSAPrivateKey = parseKey(aPKCS8Encoded512BitRSAPrivKey)    val a768BitRSAPrivateKey = parseKey(aPKCS8Encoded768BitRSAPrivKey)    fun sign(privateKey: PrivateKey, message: ByteArray): ByteArray {      val signature = Signature.getInstance("SHA256WithRSA")      signature.initSign(privateKey)      signature.update(message)      return Base64.getEncoder().encode(signature.sign())    }    val message = "foo".toByteArray()  }  @Benchmark  fun signWith512BitRSAKey(s: BenchmarkState, bh: Blackhole) {    bh.consume(s.sign(s.a512BitRSAPrivateKey, s.message))  }  @Benchmark  fun signWith768BitRSAKey(s: BenchmarkState, bh: Blackhole) {    bh.consume(s.sign(s.a768BitRSAPrivateKey, s.message))  }}

RSA keys were generated with:

$ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:512 | \    openssl pkcs8 -topk8 -nocrypt -outform DER | \    base64

This number of bits is unsafe for production use. I only chose it to get arelatively small key to prevent infinite horizontal scrolling in this blogpost. In production, use a minimal of 3072-bit RSA keys, which NIST says issafe to use through 2030.

Build the example:

$ mvn clean package

Run the example:

$ java -jar target/benchmarks.jar# JMH version: 1.37# VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7# VM invoker: /usr/lib/jvm/java-21-openjdk/bin/java# VM options: <none># Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)# Warmup: 1 iterations, 10 s each# Measurement: 1 iterations, 10 s each# Timeout: 10 min per iteration# Threads: 1 thread, will synchronize iterations# Benchmark mode: Throughput, ops/time# Benchmark: com.relentlesscoding.RSASigningMessageBenchmark.signWith512BitRSAKey# Run progress: 0.00% complete, ETA 00:00:40# Fork: 1 of 1# Warmup Iteration   1: 19.995 ops/msIteration   1: 20.506 ops/msResult "com.relentlesscoding.RSASigningMessageBenchmark.signWith512BitRSAKey":  20.506 ops/ms# JMH version: 1.37# VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7# VM invoker: /usr/lib/jvm/java-21-openjdk/bin/java# VM options: <none># Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)# Warmup: 1 iterations, 10 s each# Measurement: 1 iterations, 10 s each# Timeout: 10 min per iteration# Threads: 1 thread, will synchronize iterations# Benchmark mode: Throughput, ops/time# Benchmark: com.relentlesscoding.RSASigningMessageBenchmark.signWith768BitRSAKey# Run progress: 50.00% complete, ETA 00:00:20# Fork: 1 of 1# Warmup Iteration   1: 8.953 ops/msIteration   1: 9.193 ops/msResult "com.relentlesscoding.RSASigningMessageBenchmark.signWith768BitRSAKey":  9.193 ops/ms# Run complete. Total time: 00:00:40# ... snip ...Benchmark                                         Mode  Cnt   Score   Error   UnitsRSASigningMessageBenchmark.signWith512BitRSAKey  thrpt       20.506          ops/msRSASigningMessageBenchmark.signWith768BitRSAKey  thrpt        9.193          ops/ms

Based on this result, it looks like signing with a 512-bit RSA key is abouttwice as fast as signing with a 768-bit key on my hardware.

Migrate Nextcloud PostgreSQL 11 to 17 in a Docker Container

Sat, 18 Jan 2025 08:52:54 +0100

This post looks at how to migrate a Postgres Docker image to a newer version. Iwill dump all databases contained in a container and migrate them all into a newcontainer. The process outlined works for any Postgres container, but I discussit in the context of my Nextcloud setup.

My Setup

I have Nextcloud running in containers, with the composition detailed in adocker-compose.yml file:

Nextcloud
PostgreSQL 11
Redis
Nginx (reverse proxy)

Postgres 11 was EOL for over a year, so it was time to migrate. Mostly to getnew security patches. But newer version also incorporate new features, so a sideeffect might be (slightly) better performance.

Why Not Just Start a New Container Using the Old Data Dir?

To get the obvious out of the way, the naive approach of just running a newerPostgres image over the old data files will not work, because the data files areincompatible:

$ docker run --name pg-17 -e POSTGRES_PASSWORD=postgres \>     -v ./pg-11/data:/var/lib/postgresql/data postgres:17-alpine                                                PostgreSQL Database directory appears to contain a database; Skipping initialization2025-01-18 07:51:34.219 UTC [1] FATAL:  database files are incompatible with server2025-01-18 07:51:34.219 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 17.2.

The Plan

So we need to do a data dump and import the data into the new container (but seethe pg_upgrade alternative below). The plan is:

Stop the Nextcloud service and all its dependencies except for the database
Dump all databases into a file
Stop the old Postgres container
Spin up the new Postgres container
Import the dumped databases into the new container
Restart the Nextcloud service with its dependencies

1. Stop Nextcloud Service

$ docker compose stop nextcloud nextcloud-redis nextcloud-proxy

2. Dump All Databases Into a File

$ docker exec pg-11 pg_dumpall -U postgres -c --if-exists > pg-11.dump

The dump should be done by a database superuser as indicated by -U <user>, --username=<user>. In my case, this user is postgres. I know this, because Ihave a file db.env that defines POSTGRES_USER and POSTGRES_PASSWORD.

-c, --clean of pg_dumpall indicates that the output script will first insertDROP statements to DROP existing database objects (roles, databases) beforeinserting them. Combine this with --if-exists to make them DROP <object> IF EXISTS statements, reducing the number of errors when importing the data. Noneof this should be necessary on a new database, though, but I just want to makevery sure all the required objects are copied as the stakes are high.

Also note that in a shell, redirection is performed before the command isexecuted. In this case, the standard output is redirected to the filepg-11.dump before docker exec is executed. So the redirection affects thecurrent shell and the file will be opened on the host, not the container.

3. Stop the Running Container

$ docker compose stop nextcloud-db

4. Start New Container

We need to make sure that the new data is stored in a new location. If you runthe container as non-root (which you should), create a data directory for thenew container and chown it to the correct UID and GID used in the containerto run the postgres executable:

$ mkdir -p pg-17/data$ chown -R pguser:pggroup pg-17$ chmod -R 0700 pg-17

If you do run the container as root, you only have to specify the host directoryin docker-compose.yml (see below). The container, running as root, will beable to create the data directory anywhere on your host and then chown it tothe postgres user that is running the postgres command (by defaultpostgres).

Edit your docker-compose.yml (or adjust your -v, --volume flag to thedocker run command). It looked like this before:

services:  nextcloud-db:    image: postgres:11-alpine    container_name: pg-11    user: pguser:pggroup    volumes:      - ./pg-11/data:/var/lib/postgresql/data    environment:      - POSTGRES_USER=postgres      - POSTGRES_PASSWORD=AtriumRabidAntihero

We will change it to this:

services:  nextcloud-db:    image: postgres:17-alpine    container_name: pg-17    user: pguser:pggroup    volumes:      - ./pg-17/data:/var/lib/postgresql/data    environment:      - POSTGRES_USER=postgres      - POSTGRES_PASSWORD=AtriumRabidAntihero

Finally, start the new container:

$ docker compose up --force-recreate nextcloud-db

Unless you removed the old container, add --force-recreate to pull the newimage from Docker Hub and run it instead of restarting to old container.

5. Import the Data Into New Container

Import from inside the container:

$ docker exec -i pg-17 psql -U postgres < pg-11.dump

Use the user with superuser privileges to do the import. I do not specify adatabase name, so the default database, matching the user name, will be used.That is postgres in this case. It does not matter, as the output frompg_dumpall specifies to drop all databases and then to recreate them.

Notice that we can use the redirection in our host shell. When docker exec isexecuted, it reads from its standard input: the file pg-11.dump. For thisinput to reach psql, we need to keep standard input open by specifying -i, --interactive. If we don’t, psql does not read any of our commands, seesthat its standard input is closed, and quits. Since we are not typing commandsbut they come from a file, we do not need a prompt (pseudo-TTY) assigned (-t, --tty).

Consider adding --set ON_ERROR_STOP=1 to the command to terminate the importscript immediately when an error occurs. This is convenient because the sourceof the failure is immediately clear: it is the last attempted statement on thescreen.

Note you cannot import the dump from outside the container:

$ psql -h 127.0.0.1 -U postgres < pg-11.dump

This will prompt for your password every time the script needs to connect to anew database. The first time it will succeed, but after the ALTER ROLE postgres in the script is executed, all next authentication attempts willfail, no matter how sure you were you typed it correctly. This is explainedunder “Complications”.

Do a sanity check right after the migration:

postgres=# \l+ nextcloud                                                                                       List of databases   Name    |   Owner   | Encoding | Locale Provider |  Collate   |   Ctype    | Locale | ICU Rules |   Access privileges   |  Size   | Tablespace |                Description                 -----------+-----------+----------+-----------------+------------+------------+--------+-----------+-----------------------+---------+------------+-------------------------------------------- nextcloud | oc_stefan | UTF8     | libc            | en_US.utf8 | en_US.utf8 |        |           |                       | 206 MB  | pg_default |

6. Restart Nextcloud

$ docker compose up

Complications

Inside the Container, You Can Connect to the Database Without Credentials

If you connect to your database from inside the Docker container, either usingthe Unix socket in /var/run/postgresql or from the localhost IP address127.0.0.1, you won’t have to enter a password. This does not mean yourconfiguration is wrong or the roles are not password-protected. This is becausethe Postgres container by default trusts connections over these sockets, byvirtue of having added the following lines in/var/usr/postgresql/pg_hba.conf:

# TYPE  DATABASE        USER            ADDRESS                 METHOD# "local" is for Unix domain socket connections onlylocal   all             all                                     trust# IPv4 local connections:host    all             all             127.0.0.1/32            trust# IPv6 local connections:host    all             all             ::1/128                 trust# ...snip...host all all all scram-sha-256

This reads: allow any local user to connect as any PostgreSQL user, includingthe database superuser. Connections from outside the container, however, shouldauthenticate using the scram-sha-256 authentication method.

Connection to the Database Denied Even Though The Credentials Are Correct

After the migration is complete, we can still log in from within the container(because no authentication is required), but connections from outside thecontainer are denied:

$ psql -h 127.0.0.1 -U postgres                                      Password for user postgres:                                                                    psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL: password authentication failed for user "postgres"

This is because Postgres 11 used the md5 authentication method. This methodstores the result of hashing the password with the MD5 algorithm in thedatabase. You can see the hashes in the pg_authid table:

postgres=# select rolname, rolpassword from pg_authid where rolname = 'postgres';           rolname           |             rolpassword             -----------------------------+------------------------------------- postgres                    | md53175bce1d3201d16594cebf9d7eb3f9d

The default authentication mechanism for PostgreSQL 14+ is the (more secure)scram-sha-256. This authentication mechanism is defined in pg_hba.conf (seeabove).

What happens is that Postgres uses SCRAM-SHA-256 authentication when youauthenticate, but it cannot compare the output of that function with the hashin the database, because that was hashed with MD5. (The opposite, when md5 isset in pg_hba.conf but the result of applying SCRAM-SHA-256 is stored in thedatabase does work, though, to make the transition from the old md5 method toSCRAM-based authentication easier.)

To work around this, connect to the database with the superuser from within thecontainer (bypassing authentication), verify that the new PBKDF is in use andset new passwords for all migrated users:

$ docker exec -it pg-17 psql -U postgrespostgres=# show password_encryption; password_encryption  ---------------------  scram-sha-256  (1 row)postgres=# \passwordEnter new password for user "postgres": Enter it again:

This is Nextcloud-specific, but the application user for the Nextcloudconnection and its password can be found under/var/www/html/config/config.php.

postgres=# \password stefanEnter new password for user "stefan": Enter it again:

Obviously, as an alternative, you could also change the authentication methodin pg_hba.conf to md5 and be done. This does nothing to improve security,though, and I would not recommend since you are already need to take someaction to get it working.

Make sure you can still connect to the database:

$ psql -h 127.0.0.1 -U postgres -c 'SELECT version();'

`pg_update`

I am aware that pg_upgrade exists and can be used with Docker containers.I have not tried it, because it feels a bit more error-prone than dumping thedata and loading it in a new container.

To continue that tangent, there is a Docker image we could use topg_upgrade without having to install any Postgres libraries and executables onour host machines. I have not tried that as well, because I am not a big fan ofrunning “random” containers on my hosts.

How to Create a Shared Git Repo

Wed, 01 Jan 2025 16:25:48 +0100

In this post we will take a look at how to set up a shared Git repository. Thegoal is to have multiple different users be able to push to and pull from thisrepository. Security is provided by using Unix users and groups.

Why would you want a shared repository? The alternative, using the same user forall your repositories is bad security. If you want to grant a user access to oneof your repositories, you grant them access to all. By leveraging Unix users andgroups we can can implement least privilege: each user can only access the reposs/he has been granted access to.

Prerequisites

I will assume you have a “Git server” set up. This boils down to having a serverwith Git installed and users having access to this server over SSH. A simpleDocker container with Git and OpenSSH installed will do.

Create User and Groups

Firstly, create a user that will own all the bare Git repositories:

root@server# useradd -m git

Secondly, add the users that need access to a particular repo:

root@server# useradd -m user1root@server# useradd -m user2root@server# groupadd --users user1,user2 go-notify

You should name the group after the repository. That way, you can grant usersaccess to a repository by adding them to the group that conveniently has thesame name as the group. A name such as shared would be bad, unless you writecode for a share daemon. In this post, I will use a group go-notify toaccompany the shared go-notify repository.

Let’s check our work:

root@server# getent passwd user1 user2user1:x:12348:12348::/home/user1:/bin/shuser2:x:12349:12349::/home/user2:/bin/shroot@server# root@server# getent group go-notifygo-notify:x:60000:user1,user2

Create the Shared Repository

Let the git user create the shared repository:

git@server$ git init --bare --shared -b main go-notify.git

The --shared flag:

[specifies] that the Git repository is to be shared amongst several users.This allows users belonging to the same group to push into that repository.When specified, the config variable core.sharedRepository is set so that filesand directories under $GIT_DIR are created with the requested permissions.When not specified, Git will use permissions reported by umask(2).

Specifically, this means that the owning group will get read + write permissionson all files and directories in the shared repository. It will also set thesetgid permission on all directories, meaning that

files and subdirectories created within to inherit its group ownership, ratherthan the primary group of the file-creating process. Created subdirectoriesalso inherit the setgid bit.
– Wikipedia

Interlude: `setgid` Background

Normally, when you create a directory, the effective group of your user will bethe owning group:

$ mkdir whose-is-it$ ls -ld whose-is-itdrwxr-xr-x 2 stefan stefan 40 25 jan 14:23 whose-is-it

Things change when the setgid bit is set on a directory:

user$ mkdir shared-dirroot# groupadd somegrouproot# chgrp somegroup shared-dirroot# chmod g+s shared-dir

Note that only root can change the group ownership of a file if the effectiveuser ID is not a member of the target group. The same goes for the setgid bit.

user$ ls -ld shared-dirdrwxrwsr-x 2 user somegroup 4096 Jan 25 13:25 shared-diruser$ touch shared-dir/test.txtuser$ ls -l shared-dir-rw-r--r-- 1 user somegroup 0 Jan 25 13:33 test.txtuser$ mkdir shared-dir/subdiruser$ ls -ld shared-dir/subdirdrwxr-sr-x 2 user1 somegroup 4096 Jan 25 13:34 test/subdir

This should make clear that setgid ensures that:

New files inherit group ownership
Subdirectories maintain setgid automatically
Group ownership does not revert to user’s EGID

Continue Setting Up Shared Git Repository

Now that we understand how the --shared flag works on git init, let’s checkownership and permissions on our Git repository:

git@server:~$ ls -ld go-notify.git/drwxrwsr-x 7 git git 4096 Jan  1 15:48 go-notify.git//git@server:~$ ls -l go-notify.git/total 32-rw-rw-r-- 1 git git   21 Jan  1 15:48 HEADdrwxrwsr-x 2 git git 4096 Jan  1 15:48 branches/-rw-rw-r-- 1 git git  126 Jan  1 15:48 config-rw-rw-r-- 1 git git   73 Jan  1 15:48 descriptiondrwxrwsr-x 2 git git 4096 Jan  1 15:48 hooks/drwxrwsr-x 2 git git 4096 Jan  1 15:48 info/drwxrwsr-x 4 git git 4096 Jan  1 15:48 objects/drwxrwsr-x 4 git git 4096 Jan  1 15:48 refs/

Change group ownership of the shared repo to the shared group:

root@server# chgrp -R go-notify go-notify.git

Unless chown is invoked by a process with appropriate privileges, thesetgid is removed after a chown(1) or chgrp(1). Since we invoked it withroot, we should be fine. If you find the setgid bit removed, you set it again:

root@server# find go-notify -type d -execdir chmod g+s {} \+

Remember that only root can do this as the git user is not member of thego-notify group.

Check ownership and permissions again:

git@server:~$ ls -ld go-notify.git/drwxrwsr-x 7 git go-notify 4096 Jan  1 15:48 go-notify.git//git@server:~$ ls -l go-notify.git/total 32-rw-rw-r-- 1 git go-notify   21 Jan  1 15:48 HEADdrwxrwsr-x 2 git go-notify 4096 Jan  1 15:48 branches/-rw-rw-r-- 1 git go-notify  126 Jan  1 15:48 config-rw-rw-r-- 1 git go-notify   73 Jan  1 15:48 descriptiondrwxrwsr-x 2 git go-notify 4096 Jan  1 15:48 hooks/drwxrwsr-x 2 git go-notify 4096 Jan  1 15:48 info/drwxrwsr-x 4 git go-notify 4096 Jan  1 15:48 objects/drwxrwsr-x 4 git go-notify 4096 Jan  1 15:48 refs/

Push With Remote User

Remote user user1 can now create a file and push it:

user1@pc$ git init -b main repouser1@pc$ cd repouser1@pc$ echo 'This is user1' > user1.txtuser1@pc$ git add user1.txtuser1@pc$ git commit -m 'chore: initial commit by user1'user1@pc$ git remote add origin ssh://user1@server:/home/git/go-notify.git/

Now we can push:

user1@pc$ git push -u origin mainuser1@127.0.0.1's password:fatal: detected dubious ownership in repository at '/home/git/go-notify.git'To add an exception for this directory, call:        git config --global --add safe.directory /home/git/go-notify.gitfatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists.

Designate Safe Directories

Oops. It turns out that Git is particular about which repositories (directoriesreally) it will fetch from and push to. git-config(1) gives the reason:

safe.directory
These config entries specify Git-tracked directories that areconsidered safe even if they are owned by someone other than the current user.By default, Git will refuse to even parse a Git config of a repository ownedby someone else, let alone run its hooks, and this config setting allows usersto specify exceptions, e.g. for intentionally shared repositories (see the--shared option in git-init(1)).

Since we are creating a shared directory, for every user that is allowed tointeract with it, set it as a safe.directory on the server:

root@server# su -l user1 -c 'git config --global \    --add safe.directory /home/git/go-notify.git'root@server# su -l user2 -c 'git config --global \    --add safe.directory /home/git/go-notify.git'

Note that it will not work if you put a trailing forward slash aftergo-notify.git.

Alternatively, trust the directory system-wide:

root@server# git config --system \    --add safe.directory /home/git/go-notify.git

This will write to the system-wide configuration at /etc/gitconfig.

Now we can push:

user1@pc$ git push -u origin mainEnumerating objects: 3, done.Counting objects: 100% (3/3), done.Writing objects: 100% (3/3), 403 bytes | 403.00 KiB/s, done.Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)To ssh://server:/home/git/go-notify.git * [new branch]      HEAD -> main branch 'main' set up to track 'origin/main'.

Another User Can Clone and Push As Well

user2 can clone the repo and then push its own commit:

user2@pc:~$ git clone ssh://user2@server:/home/git/go-notify.gituser2@pc:~$ cd go-notify/user2@pc:~$ cat > user2.txtFile written by user2^Duser2@pc:~$ git add user2.txtuser2@pc:~$ git commit -m 'chore: first commit by user2'user2@pc:~$ git push

Alternatives

Using git init --shared is one way of achieving shared repositories. As thisleverages standard Unix groups, this is pretty simple. Another way is leveragingACLs, as discussed in a previous blog post.This is more complex, but also more flexible.

Docker Volume Permissions: Mismatch Between Host UID and Container UID

Mon, 30 Dec 2024 15:40:50 +0100

When you have a Docker container with a user that has a different UIDfrom the one on your host, mounting a volume and accessing the files canresult in permission issues.

Let’s say the UID of the Docker user is 12345. The files on the hostare owned by UID 1000. This means the permissions do not hash and thecontainer user cannot access the files.

One solution is to use ACLs to grant the container user access.

Prerequisites

After reading the Arch Linux wiki, check if you filesystem supportsACLs:

# tune2fs -l /dev/sdXY | grep "Default mount options:"Default mount options:    user_xattr acl

New Files Inherit Group

$ find ./somedir -type d -execdir chmod g+s {} \+

First, we set the SGID bit on all directories in the volume. Now, newlycreated files inherit the group ownership of the directory. When theuser 12345 creates a new file or directory inside the Dockercontainer, the host user 1000 will still be able to access thosefiles.

Set Inherited Default ACL on Directories

$ find ./somedir -type d -execdir setfacl -dm u:12345:rwX {} \+

Next, we set default ACLs on all directories in the volume. Newlycreated files will inherit this default ACL. If the default ACL hasuser:12345:rwx specified and we create a new file with touch, whichuses a mode of 00666, the resulting file will have an ACL ofuser:12345:rw-. This is because no permissions can be present that arenot specified in the mode parameter of the open system call.acl(5) explains this in more detail.

Set ACL on All Files and Directories

$ find ./somedir -execdir setfacl -m u:12345:rwX {} \+

Finally, we set rw permissions on all files and directories. The Xis handy, it sets

execute/search only if the file is a directory or already has executepermission for some user

So it will only set search permissions on directories, which need it sothe Docker user can descend into them, or on files that are alreadyexecutable.

Postgres Data Types, Sizes and Alignment

Thu, 08 Aug 2024 08:12:55 +0200

How much storage does a single row in a Postgres table take?

Page Format

A page in a default Postgres installation is 8 KiB in size. Each page begins with 24 bytes of PageHeaderData(metainformation). This metainformation is followed by ItemIds, eachcontaining the offset (from the start of this page) and the length ofthe row it points to. An ItemId is 4 bytes. ItemIds are put afterthe PageHeaderData, the items (rows) themselves are stored from theend of the page backwards.

Row Sizes

Each row in a table has a fixed-size, 23-byte header, followed by anoptional “null bitmap”. When NULLable types are used in a tabledefinition, the bitmap contains a bit per field: 0 if the content ofthis row’s field is not NULL, 1 if it is. When the definition has 9+fields (so more bits than can be contained in a single byte), it growsto 2 bytes. The user data, however, only starts at a fixed MAXALIGNboundary (8 bytes on a 64-bit machine, 4 on 32-bit).

=# select pg_column_size(row(null, null, null, null, null, null, null, null)); pg_column_size----------------             24=# select pg_column_size(row(null, null, null, null, null, null, null, null, null)); pg_column_size----------------             32

(The row() constructor creates a row on an anonymous recordtype.)

If we have 8 NULLable values, the fields fit into a single-byte nullbitmap. If we have more than 8, more bytes are required. In this case, 2bytes. On a 64-bit machine, user data starts at an 8-byte boundary. Sothe user data starts at 23 + 2 + 7 = 32 bytes.

Data Sizes

There are two types of type lengths: fixed-length and variable-lengthtypes. A smallint (= int2), for example, is a fixed-length typetaking 2 bytes of storage. A text type is a variable-length type.Since rows cannot span pages, when the data of a variable-length typeexceeds 2 KiB, it can be stored out-of-line (in another table). In thatcase, instead of the value itself, a pointer to that data is stored.(See the example later on.)

Data Alignment

Each data type has alignment requirements. You can view theserequirements:

=# select typname, typlen, typalign from pg_type where typname = 'int2'; typname | typlen | typalign ---------+--------+---------- int2    |      2 | s

For a smallint/int2, we see that typalign is s. This indicatesalignment on a short boundary, which equals 2 bytes on most machines.Consequently, if an int2 field is defined after a field that takes anodd amount of storage, a padding byte is inserted:

=# select pg_column_size(row(true)); pg_column_size----------------             25=# select pg_column_size(row(true, 2::int2)); pg_column_size----------------             28

In the first case: 23 bytes of fixed-size header + 1 byte padding toalign to MAXALIGN + 1 byte for the boolean.

If we throw an int2 in the mix after this boolean, we see that thetotal size increases by 3: 1 padding byte and 2 bytes to store theint2 value.

TOAST

Types with flexible-storage requirements use the internal varlenatype. Depending on the length of the data they store, they have a 1- or4-byte header. Their actual values may be stored in-line (meaning in therow itself) or out-of-line (meaning in another table). Wherever theactual data may live, it can also be compressed. The technique fordealing with arbitrary-length fields is called TOAST.

Values with a single-byte header are not aligned on any particularboundary:

=# select pg_column_size(row(true, 'foo'::text)); pg_column_size----------------             29

After the 24-byte row header, we see 1 byte for the boolean, and then 1

3 = 4 bytes for the text type.

In the 1-byte header, when the most-significant is set, the remaining 7bits indicate the length of the data in bytes (including this header).It therefore can indicate up to 126 bytes of data (2^7 - 1 header byte).

=> select pg_column_size(row(true, repeat('b', 127))); pg_column_size----------------            159

Here we see that once the text type exceeds 126 bytes 2 things happen:

the type now requires a header of 4 bytes; and
it now needs to align on a integer (4-byte) boundary.

When we calculate ourselves: 24 bytes fixed size header, 1 byte to storethe boolean, 3 padding bytes to make sure the 4-byte text headerstarts on a 4-byte boundary, 4 bytes for the text header and finally127 bytes to store the actual string.

TOAST Storage Options

TOAST is only used when the data-to-be-stored exceedsTOAST_TYPLE_THRESHOLD (2 KiB on default installations).

You can set storage for a TOASTable column with ALTER TABLE ... SET STORAGE.

There are 4 strategies for TOASTable columns:

PLAIN (no compression or out-of-line storage is done)
EXTENDED (attempt compression, then out-of-line storage)
EXTERNAL (attempt only out-of-line storage)
MAIN (attempt only compression)

The default storage strategies for types can be seen in the pg_typesystem catalog:

=# select typname, typlen, typalign, typstorage from pg_type where typname = 'int8'; typname | typlen | typalign | typstorage---------+--------+----------+------------ int8    |      8 | d        | p=# select typname, typlen, typalign, typstorage from pg_type where typname = 'text'; typname | typlen | typalign | typstorage---------+--------+----------+------------ text    |     -1 | i        | x

An int8 takes a fixed 8 bytes and is not TOASTable as indicated byboth the typlen and the typstorage (p stands for PLAIN).

A text field is a variable-length type. This is indicated by thenegative typlen of -1. The storage strategy is x, which stands forEXTENDED. If you have a value that is > 2 KiB, it will either becompressed or moved to a another table for storage:

=# create table t (t text);CREATE TABLE=# insert into t values (repeat('a', 1024));INSERT 0 1=# insert into t values (repeat('a', 2048));INSERT 0 1=# select pg_column_size(t) from t t; pg_column_size----------------           1028             35

A text value of length 1024 is stored in-line after a 4-byte header. Atext value whose storage (including header) exceeds 2 KiB is stored ina TOAST table. In its place, a pointer to the data is inserted in ourtable. The documentation says the pointer data is 18 bytes long(storing the toast table OID, chuck OID, logical size, physical size andthe compression method used, if any), but here we see that only 11 (35 -24) bytes were used. That is something to figure out for me :)

Find Total Size On Disk of a Table

To find the total size of a table, including any indexes and TOASTtables (where oversized data of variable-length types are stored):

=# select pg_total_relation_size('t'); pg_total_relation_size------------------------                  16384

The same can be achieved when using the psql CLI client:

=# \dt+ t                                   List of relations Schema | Name | Type  | Owner | Persistence | Access method | Size  | Description--------+------+-------+-------+-------------+---------------+-------+------------- public | t    | table | owner | permanent   | heap          | 16 kB |

There are more handy functions to calculate different types of objectsavailable.

Remove All CAs From Linux Trust Store

Fri, 12 Jul 2024 19:39:01 +0200

On Security Now! podcast #951, Steve told us that 7organizations in the PKI are responsible for minting upwards of 99% ofall existing certificates. (Possible original source.)

His suggestion was that we can remove all CAs except for those 7 andstill be able to browse the web normally. I wanted to try that, so Idisabled almost all CAs in Firefox by hand (about:preferences#privacyand then “View Certificates…”). After allowing some other CAs that Iencountered on web sites and that I thought were trustworthy, it workedbeautifully.

I wanted to do the same for curl, wget and other programs.

First I read the Arch Linux Wiki article on TLS certificates. Apparently, libraries can make use of the PKCS #11interface exposed by Arch. I searched for it, but could not readily findany programs that use this interface. From experimenting, I found thatcurl, wget, openssl and docker all obtain their root store fromfiles in /etc/ca-certificates/.

My tactic is: disallow all CAs, then go about my day, and enable thosethat I encounter and that seem trustworthy.

1. Check if you can securely connect to a remote host

$ H=nos.nl; echo Q | \>   openssl s_client -verify_return_error -servername $H $H:443Connecting to 23.50.131.144CONNECTED(00000003)depth=2 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CAverify return:1depth=1 C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1verify return:1depth=0 C=BE, L=Antwerpen, O=DPG Media Services NV, CN=*.nu.nlverify return:1... snip ...

2. Disable All CAs

$ sudo ln /etc/ca-certificates/extracted/cadir/*.pem \>   /etc/ca-certificates/trust-source/blocklist/$ sudo /usr/bin/update-ca-trust

3. Verify that remote host is now untrusted

$ H=nu.nl; echo Q | \>   openssl s_client -verify_return_error -servername $H $H:443Connecting to 23.50.131.156CONNECTED(00000003)depth=1 C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1verify error:num=20:unable to get local issuer certificate40575E1A63700000:error:0A000086:SSLroutines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091:---Certificate chain 0 s:C=BE, L=Antwerpen, O=DPG Media Services NV, CN=*.nu.nl   i:C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256   v:NotBefore: Jun  2 00:00:00 2024 GMT; NotAfter: Jun  4 23:59:59 2025 GMT 1 s:C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1   i:C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256   v:NotBefore: Apr 14 00:00:00 2021 GMT; NotAfter: Apr 13 23:59:59 2031 GMT... snip ...

4. Allow CA if deemed trustworthy

If we look at the output under 3., we see that the issuer of theintermediate certificate (s:C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1) is DigiCert. To trust this root again:

$ find /etc/ca-certificates/trust-source/blocklist/ \>   -iname '*digicert*' -print -delete./DigiCert_High_Assurance_EV_Root_CA.pem./DigiCert_Global_Root_G2.pem./DigiCert_TLS_RSA4096_Root_G5.pem./DigiCert_Trusted_Root_G4.pem./DigiCert_Assured_ID_Root_G2.pem./DigiCert_Assured_ID_Root_G3.pem./DigiCert_Global_Root_G3.pem./DigiCert_TLS_ECC_P384_Root_G5.pem./DigiCert_Assured_ID_Root_CA.pem./DigiCert_Global_Root_CA.pem$ /usr/bin/update-ca-trust

Sometimes, an application caches the blocklisted certs. In that case, you needto restart the application. Notably, the Docker daemon needs to be restartedwhenever a change to the certificate store was made.

5. Check if domain gets validated again

$ H=nu.nl; echo Q | openssl s_client -verify_return_error -servername $H $H:443Connecting to 23.50.131.156CONNECTED(00000003)depth=2 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CAverify return:1depth=1 C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1verify return:1depth=0 C=BE, L=Antwerpen, O=DPG Media Services NV, CN=*.nu.nlverify return:1... snip ...

Cookies

Sat, 25 May 2024 13:53:55 +0200

Under what circumstances cookies are accepted or rejected by user agentsand send back to web servers can be a bit confusing. Let’s experiment abit.

TL;DR

Actually, it is pretty simple:

The origin domain is the domain you’re visiting. An origin domain canset cookies for its own domain and any higher-level domains. A clientwill send cookies for the origin domain and any higher-level domains.Conversely, a higher-level domain can read nor write cookies forsubdomains.

So: example.com can read and write cookies for example.com but notsub.example.com. sub.example.com can read and write cookies forsub.example.com and example.com, but not for its siblingsub2.example.com. Top-level domains (TLDs) are always off-limit foreveryone.

Test Drive

We’re going to prove all the cases (and some special ones) to ourselvesusing our favorite user agent: curl(1)! Please note that Iwon’t be showing all of the command-line output everywhere to preventthis post from becoming too large.

Setup

Add the following lines to /etc/hosts:

127.0.0.1example.com sub1.example.com sub2.sub1.example.com127.0.0.1other.com sub1.other.com sub2.sub1.other.com

Create a self-signed certificate. Every curl invocationshould specify --cacert <cert> so curl will accept and verify theself-signed certificate.

Run the following Golang program:

cookie.go

 // expirements with setting cookies package main import (     "fmt"     "log"     "net/http" ) var counter = 0 var inc = func() string {     counter++     return fmt.Sprintf("%d", counter) } func main() {     http.HandleFunc("/other-domain", handleOtherDomain)     http.HandleFunc("/tld", handleTLD)     http.HandleFunc("/subdomain", handleSubdomain)     http.HandleFunc("/domainless", handleNoDomain)     http.HandleFunc("/host-prefix-secure", handleHostPrefixSecure)     http.HandleFunc("/host-prefix-not-secure", handleHostPrefixNotSecure)     http.HandleFunc("/secure", handleSecure)     http.HandleFunc("/not-secure", handleNotSecure)     go func() {         log.Fatal(http.ListenAndServeTLS("127.0.0.1:443", "cert.pem", "key.pem", nil))     }()     go func() {         log.Fatal(http.ListenAndServe("127.0.0.1:80", nil))     }()     select {} } // handleOtherDomain tries to set cookie for different domain func handleOtherDomain(w http.ResponseWriter, _ *http.Request) {     http.SetCookie(w, &http.Cookie{         Name:   "other",         Value:  inc(),         Domain: "other.com",     }) } // handleTLD tries to set a "supercookie" for a TLD func handleTLD(w http.ResponseWriter, _ *http.Request) {     http.SetCookie(w, &http.Cookie{         Name:   "tld",         Value:  inc(),         Domain: "com",     }) } // handleSubdomain sets 3 cookies for parent and 2 subdomains func handleSubdomain(w http.ResponseWriter, _ *http.Request) {     // can be set on parent or any subdomain of parent     http.SetCookie(w, &http.Cookie{         Name:   "parent",         Value:  inc(),         Domain: "example.com",     })     // can be set from sub1 domain or any subdomain of sub1     http.SetCookie(w, &http.Cookie{         Name:   "sub1",         Value:  inc(),         Domain: "sub1.example.com",     })     // can be set from sub2 domain or any subdomain of sub2     http.SetCookie(w, &http.Cookie{         Name:   "sub2",         Value:  inc(),         Domain: "sub2.sub1.example.com",     }) } // handleNoDomain set cookie w/ no Domain attribute: can be set on origin // domain only and will be sent only to origin domain, not subdomains func handleNoDomain(w http.ResponseWriter, r *http.Request) {     http.SetCookie(w, &http.Cookie{         Name:  fmt.Sprintf("domainless-%s", r.Host),         Value: inc(),     }) } // handleHostPrefixSecure sets "domain-locked" cookie: has Secure attr, no // Domain and is only accepted when sent over secure connection func handleHostPrefixSecure(w http.ResponseWriter, _ *http.Request) {     http.SetCookie(w, &http.Cookie{         Name:   "__Host-host",         Value:  inc(),         Secure: true,     }) } // handleHostPrefixNotSecure tries to set __Host- cookie w/o Secure attr // these "domain-locked" cookies should have Secure attr, no Domain and // sent over secure connection func handleHostPrefixNotSecure(w http.ResponseWriter, _ *http.Request) {     http.SetCookie(w, &http.Cookie{         Name:   "__Host-host",         Value:  inc(),         Secure: false,     }) } // handleSecure tries to set secure cookie // should only be accepted (and send) by clients when sent over secure connection func handleSecure(w http.ResponseWriter, _ *http.Request) {     http.SetCookie(w, &http.Cookie{         Name:   "secure",         Value:  inc(),         Domain: "example.com",         Secure: true,     }) } // handleNotSecure sets cookie with name "secure" but w/o Secure attr set func handleNotSecure(w http.ResponseWriter, _ *http.Request) {     http.SetCookie(w, &http.Cookie{         Name:   "secure",         Value:  inc(),         Domain: "example.com",         Secure: false,     }) }

Let’s begin with an empty cookie jar:

$ > jar

Let’s get “setting a cookie for another domain” out of the way first:

$ curl -v -b jar -c jar https://example.com/other-domain... snip ...> GET / HTTP/2> Host: example.com> User-Agent: curl/8.5.0> Accept: */*> < HTTP/2 200 * skipped cookie with bad tailmatch domain: other.com< set-cookie: other=other; Domain=other.com< content-length: 0< date: Sat, 25 May 2024 12:15:58 GMT

curl informs us that example.com can’t set a cookie for other.com.Check.

$ curl -v -b jar -c jar https://example.com/tld... snip ...* cookie 'tld' dropped, domain 'example.com' must not set cookies for 'com'< set-cookie: tld=tld; Domain=com... snip ...

We can’t set a cookie for .com or any other TLD. Another check.

$ curl -v -b jar -c jar https://example.com/subdomains... snip ...* Added cookie parent="parent" for domain example.com, path /, expire 0< set-cookie: parent=parent; Domain=example.com* skipped cookie with bad tailmatch domain: sub1.example.com< set-cookie: sub1=sub1; Domain=sub1.example.com* skipped cookie with bad tailmatch domain: sub2.sub1.example.com< set-cookie: sub2=sub2; Domain=sub2.sub1.example.com... snip ...

Only the cookie set for itself is accepted. Cookies for subdomains arerejected. Another check.

Let’s examine the cookie jar at this point:

$ tail -n +5 jar | column -t.example.com  TRUE  /  FALSE  0  parent  parent

We notice that the cookie with name parent was accepted and put in thejar. It’s a cookie that the client will send to subdomains as we’llsee in a bit. This is indicated by the leading dot .example.com andthe TRUE in the second column. The jar file format is explainedhere.

Do User Agents Send Cookies for Parents to Subdomains?

Let’s make a request to sub1.example.com:

$ curl -v -b jar -c jar https://sub1.example.com/subdomains... snip ...> GET / HTTP/2> Host: sub1.example.com> User-Agent: curl/8.5.0> Accept: */*> Cookie: parent=parent> < HTTP/2 200 * Replaced cookie parent="parent" for domain example.com, path /, expire 0< set-cookie: parent=parent; Domain=example.com* Added cookie sub1="sub1" for domain sub1.example.com, path /, expire 0< set-cookie: sub1=sub1; Domain=sub1.example.com* skipped cookie with bad tailmatch domain: sub2.sub1.example.com< set-cookie: sub2=sub2; Domain=sub2.sub1.example.com

The client sends a cookie set on the parent domain to the subdomain.
The client accepts a cookie with the parent domain, replacing theprevious cookie, and for the origin domain.
The cookie for a subdomain, sub2.sub1.example.com is rejected.

To really drive this point home, let’s make a request tosub2.sub1.example.com:

$ curl -v -b jar -c jar https://sub2.sub1.example.com/subdomains> GET / HTTP/2> Host: sub2.sub1.example.com> User-Agent: curl/8.5.0> Accept: */*> Cookie: sub1=sub1; parent=parent> < HTTP/2 200 * Replaced cookie parent="parent" for domain example.com, path /, expire 0< set-cookie: parent=parent; Domain=example.com* Replaced cookie sub1="sub1" for domain sub1.example.com, path /, expire 0< set-cookie: sub1=sub1; Domain=sub1.example.com* Added cookie sub2="sub2" for domain sub2.sub1.example.com, path /, expire 0< set-cookie: sub2=sub2; Domain=sub2.sub1.example.com

Now the jar contains:

$ tail -n +5 jar | column -t.sub2.sub1.example.com  TRUE  /  FALSE  0  sub2    sub2.sub1.example.com       TRUE  /  FALSE  0  sub1    sub1.example.com            TRUE  /  FALSE  0  parent  parent

Are Subdomain Cookies Send to the Parent Domain?

What cookies are send to example.com?

$ curl -v -b jar -c jar https://example.com> GET / HTTP/2> Host: example.com> User-Agent: curl/8.5.0> Accept: */*> Cookie: parent=parent... snip ...

Only the ones set for the origin domain, not for any subdomains.

Special Cases

A cookie without a Domain attribute will be accepted for the origindomain only. It will not be sent with requests to subdomains.

$ # empty jar$ > jar$ curl -v -b jar -c jar https://example.com/domainless> GET / HTTP/2> Host: example.com> User-Agent: curl/8.5.0> Accept: */*> < HTTP/2 200 * Added cookie domainless-example.com="domainless-example.com" for domain example.com, path /, expire 0< set-cookie: domainless-example.com=domainless-example.com

Check the jar:

$ tail -n +5 jar | column -texample.com  FALSE  /  FALSE  0  domainless-example.com  domainless-example.com

Notice the missing dot in the cookie jar and the FALSE in the secondcolumn. This cookie is not send to subdomains:

$ curl -v -b jar -c jar https://sub1.example.com> GET / HTTP/2> Host: sub1.example.com> User-Agent: curl/8.5.0> Accept: */*> < HTTP/2 200 * Added cookie domainless-sub1.example.com="domainless-sub1.example.com" for domain sub1.example.com, path /, expire 0< set-cookie: domainless-sub1.example.com=domainless-sub1.example.com... snip ...

Mozilla writes that there are certain prefixes that are respected byHTTP user agents. Setting a cookie that has a name startingwith __Host- will be accepted only if (1) the Secure attribute hasbeen set, (2) it was sent from a secure origin, (3) the Domainattribute is missing and Path is set to /:

$ curl -v -b jar -c jar https://example.com/host-prefix-secure> GET / HTTP/2> Host: example.com> User-Agent: curl/8.5.0> Accept: */*> < HTTP/2 200 * Added cookie __Host-host="__Host-host" for domain example.com, path /, expire 0< set-cookie: __Host-host=__Host-host; Secure

Let’s see what happens when it is sent over an insecure channel:

$ # empty jar first$ > jar$ curl -v -b jar -c jar http://example.com/host-prefix-secure> GET / HTTP/1.1> Host: example.com> User-Agent: curl/8.5.0> Accept: */*> < HTTP/1.1 200 OK< Set-Cookie: __Host-host=__Host-host; Secure... snip ...

The jar stays empty:

$ tail -n +5 jar | column -t$

Sending over secure channel but not setting Secure attribute:

$ curl -v -b jar -c jar https://example.com/host-prefix-not-secure> GET / HTTP/2> Host: example.com> User-Agent: curl/8.5.0> Accept: */*> < HTTP/2 200 < set-cookie: __Host-host=__Host-host... snip ...$ tail -n +5 jar | column -t$

Another prefix is __Secure-. Cookies sent with this prefix are onlyaccepted when it’s marked Secure and send over a secure channel. Soit’s weaker than __Host-.

The reason these prefixes are used is that normally a server can’t besure that a cookie was set from a secure origin or even what exactorigin domain set the cookie. Remember that subdomains can set cookiesfor parent domains. A subdomain can insecurely set a cookie that theparent domain also sets with Secure and therefore expects to besecured.

The `Secure` Attribute

Clients reject cookies marked Secure if received over an insecurechannel:

$ curl -v -b jar -c jar http://example.com> GET / HTTP/1.1> Host: example.com> User-Agent: curl/8.5.0> Accept: */*> < HTTP/1.1 200 OK< Set-Cookie: secure=secure; Domain=example.com; Secure$ tail -n +5 jar | column -t$

Over a secure channel, this cookie is accepted. However, if a subdomainoverrides this cookie and leaves out the Secure attribute:

$ curl -v -b jar -c jar https://sub1.example.com/not-secure> GET / HTTP/2> Host: sub1.example.com> User-Agent: curl/8.5.0> Accept: */*> Cookie: secure=secure> < HTTP/2 200 * Replaced cookie secure="secure" for domain example.com, path /, expire 0< set-cookie: secure=secure; Domain=example.com$ tail -n +5 jar | column -t.example.com  TRUE  /  FALSE  0  secure  secure

Notice that the 4th field is FALSE, indicating thiscookie is not Secure. Now, when we make a request to the parentdomain:

$ curl -v -b jar -c jar https://example.com> GET / HTTP/2> Host: example.com> User-Agent: curl/8.5.0> Accept: */*> Cookie: secure=secure> < HTTP/2 200 ... snip ...

So that’s why __Host- and __Secure- prefixes exist.

Source Code

You can get the source code used in this post from Source Hut:https://git.sr.ht/~neftas/cookie.

Create Self-Signed TLS Cert With Wildcard

Sat, 25 May 2024 12:17:05 +0200

Let’s create a self-signed certificate that contains a wildcard SubjectAlternative Name (SAN).

$ openssl req -x509 \    -subj "/C=NL/L=Utrecht/O=Relentless Coding/CN=*.example.com" \    -addext "subjectAltName = DNS:example.com, DNS:*.example.com" \    -newkey rsa:2048 \    -noenc \    -keyout key.pem \    -out cert.pem

-addext allows us to add X509 extensions to the certificate from thecommand line. In older versions of openssl(1) you needed to use aconfiguration file (--config openssl.cnf) to achieve the same thing.(See x509v3_config(5) for more information about X509 extensions.)

Note we can add multiple DNS names to the SAN field by separating themwith commas. This is different from the way this information is providedin the configuration file.

Also note that -noenc replaced the -nodes flag in openssl version3. They both indicate that the private key is not to be encrypted with apassphrase.

Copy key.pem and cert.pem to where the server reads TLS certs andrestart it.

Add entry to /etc/hosts:

127.0.0.1example.com sub.example.com

Use with curl:

$ curl --cacert cert.pem https://sub.example.com

Notice we use --cacert <cert>. This way, the server can still presenta self-signed certificate, and we still get the benefit from allcertificate checks curl does, such as domain checking. (The -k, --insecure flag disables all those checks, so it will accept anycertificate, including a certificate for other.com when connecting toexample.com, which is not always what we want.)

Overwrite File Without Truncating

Mon, 01 Apr 2024 09:44:23 +0200

In Bash, when you open a file for writing, the file gets truncated.However, there is a way to open a file for both reading and writing atthe same time.

Who hasn’t made the mistake of creating a process that tries to bothread and write to the same file?

$ wc -c file17 file$ sed s/foo/FOO/ <file >file$ wc -c file0

Oops. That’s why for sed(1) in particular you use the-i[SUFFIX], --in-place[=SUFFIX] switch, and in general can usesponge(1) from moreutils (e.g. grep foo <file | sponge >file).

But you can also open a file for reading and writing, by using the[n]<>word syntax. This is described in bash(1) under“REDIRECTION” > “Opening File Descriptors for Reading and Writing”:

The redirection operator
    [n]<>word
causes the file whose name is the expansion of word to beopened for both reading and writing on file descriptor n, oron file descriptor 0 if n is not specified. If the file doesnot exist, it is created.

Test drive:

$ cd $(mktemp -d)$ ls$ cat > testHello, world!^D$ cat >> testAnother line^D$ cat 1<> testG'day^D$ cat testG'day, world!Another line

Bash's exec Replaces Shell

Sat, 30 Mar 2024 10:44:10 +0100

This is a quick look at Bash’s exec builtin with a simple example.

bash(1) explains exec [command]:

If command is specified, it replaces the shell. No new process iscreated.

This is mainly useful for wrapper scripts, where wedo not want to have a Bash process lingering around. Take this scriptnamed test:

#!/bin/bashsleep inf

When run as ./test, the current shell creates a new Bash instance (theinterpreter specified in the first line of the script with the shebang#!/bin/bash), and then creates a new process for the sleep command.So, we have 2 processes running: a Bash and a sleep process:

$ ps -o pid,ppid,cmd -T    PID    PPID CMD   2443     969 -bash   5348    2443 /bin/bash ./test   5349    5348 sleep inf   5358    2443 ps -o pid,ppid,cmd -T

When we background ./test with Ctrl-Z and use ps(1) with the-T flag, it prints the processes associated with the current terminalsession. PID 2443 is the current terminal. We see that 2443 spawned/bin/bash ./test (PID 5348) which in its turn begat sleep inf (PID5349).

Now we modify the script to use exec to execute sleep:

#!/bin/bashexec sleep inf

Again, the current shell executes a new Bash process. But instead ofit creating a new sleep process, the sleep process replaces it. Now weonly have single process (sleep) running:

$ ps -o pid,ppid,cmd -T    PID    PPID CMD   2443     969 -bash   5399    2443 sleep inf   5407    2443 ps -o pid,ppid,cmd -T

Here, we see that sleep inf has as its parent the shell that invoked./test: it has replaced /bin/bash ./test.

Commands in Pipelines Are Executed in Subshells

Sat, 30 Mar 2024 10:44:10 +0100

Let’s have a look at another one of those gotchas that can bite youwhile scripting in Bash: processes that are part of a pipeline all runin subshells.

What would be result of executing the following Bash script?

var=fooecho bar | read varecho $var

If you said “foo”, you would be right. For me, this was reallycounter-intuitive. What is going on?

As always, bash(1) has the answer. It explains under“SHELL GRAMMAR” > “Pipelines”:

Each command in a multi-command pipeline, where pipes are created, isexecuted in a subshell, which is a separate process.

And also, under “COMMAND EXECUTION ENVIRONMENT”:

Changes made to the subshell environment cannot affect the shell’sexecution environment.

That means a pipeline can actually be represented as:

<new, separate process> | <another new, separate process>

Let’s prove this to ourselves. Under “PARAMETERS” > “SpecialParameters”, Bash’s man page explains that Bash expands $$:

[…] to the process ID of the shell. In a subshell, it expands to theprocess ID of the current shell, not the subshell.

$ echo $$; >&2 echo $$ | echo $$244324432443

To get the PID of the subshell, Bash provides BASHPID (documentedunder “PARAMETERS” > “Shell Variables”):

Expands to the process ID of the current bash process. This differsfrom $$ under certain circumstances, such as subshells that do notrequire bash to be re-initialized.

$ echo Parent: $BASHPID; >&2 echo Sub 1: $BASHPID | echo Sub 2: $BASHPIDParent: 2443Sub 2: 10599Sub 1: 10598

Here, the parent shell has PID 2443, and the two subshells have 10599and 10598 respectively.

Create Persistent Connections to Processes Using FIFOs

Tue, 26 Mar 2024 09:21:43 +0100

In this post, we are going to make a Bash script interact with an HTTPserver and have multiple exchanges using the same connection byleveraging named pipes/FIFOs.

My first, naive approach was to first create named pipes i (for“input”) and o (for “output”). I ran openssl s_client with its stdinreading from i and its stdout writing to o and backgrounded it.Then, I used the shell builtin printf and cat to write and read fromi and o respectively:

#!/bin/bashmkfifo i oh=bol.comopenssl s_client -servername $h $h:443 <i >o &printf '%s\r\n' 'GET / HTTP/1.1' 'Host: bol.com' '' >icat <o

This results in the following (output is truncated for brevity):

CONNECTED(00000003)---Certificate chain 0 s:CN=www.bol.com   i:C=US, O=Let's Encrypt, CN=R3   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256   v:NotBefore: Mar 17 10:19:22 2024 GMT; NotAfter: Jun 15 10:19:21 2024GMT 1 s:C=US, O=Let's Encrypt, CN=R3   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025GMT---Server certificate-----BEGIN CERTIFICATE-----... snip ...

In Bash, redirections (e.g. reading from or writing to a named pipe) areprocessed before starting a command. Therefore, Bash blocks while tryingto open the named pipes until both the reading and writing ends areconnected. That means that openssl will not be executed until it hassome other process writing to its input pipe and reading from its outputpipe.

(We can prove this to ourselves by doing a process listing:

$ h=bol.com; openssl s_client -servername $h $h:443 <i >o &$ pgrep openssl; echo $?1

Even when writing to the input pipe, but not reading yet from the outputpipe, the process is not started yet:

$ printf '%s\r\n' 'GET / HTTP/1.1' 'Host: bol.com' '' >i$ pgrep openssl; echo $?1

So, only when some other process starts reading from its output willopenssl actually start.

Trying to run strace(1) on cat <o was an eye-opener forme:

$ strace cat <o

This actually blocks any strace output until the writing end iswritten to, showing clearly that “redirection is done before any processis started”.

By having a new Bash process execute this code, we can bypass this. Say,we have a file test:

#!/bin/bashcat <i

We execute this with strace -f -yy bash test. When that blocks atopenat (see code listing below), we execute /bin/echo foo >i fromanother terminal window:

... snip ...[pid  4497] openat(AT_FDCWD</tmp/tmp.EwoMhpZw3f>, "i", O_RDONLY) = 3</tmp/tmp.EwoMhpZw3f/i>[pid  4497] dup2(3</tmp/tmp.EwoMhpZw3f/i>, 0</dev/pts/8<char 136:8>>) = 0</tmp/tmp.EwoMhpZw3f/i>[pid  4497] close(3</tmp/tmp.EwoMhpZw3f/i>) = 0[pid  4497] execve("/usr/bin/cat", ["cat"], 0x5c51da8fdb20 /* 52 vars */) = 0... snip ...

We see that the system call execve on /usr/bin/cat was only invokedafter the writing end of the pipe was opened.

Likewise, when straceing the script containing /bin/echo foo >i, wesee:

[pid  9261] openat(AT_FDCWD</tmp/tmp.wVg9PTBQxY>, "i", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3</tmp/tmp.wVg9PTBQxY/i>[pid  9261] dup2(3</tmp/tmp.wVg9PTBQxY/i>, 1</dev/pts/6<char 136:6>>) = 1</tmp/tmp.wVg9PTBQxY/i>[pid  9261] close(3</tmp/tmp.wVg9PTBQxY/i>) = 0[pid  9261] execve("/bin/echo", ["/bin/echo", "foo"], 0x5b99e9dfeb90 /* 52 vars */) = 0

/bin/echo is only execved after opening i.)

In our HTTP case, the writing happens by printf >i, the reading bycat <o.

However, the output reader only sees TLS handshake information. Theconnection to the remote server is closed. Although strace confirmsthat openssl did sent the request to the remote server, we did not seeany HTTP response in the output. Why not? According topipe(7):

If all file descriptors referring to the write end of a pipe havebeen closed, then an attempt to read(2) from the pipe will seeend-of-file (read(2) will return 0).

That means that openssl has seen an EOF when it tried to read fromi, and terminated itself immediately(openssl-s_client(1) provides -ign-eof to prevent that,BTW).

(To empirically show that the last writer closes the writing end of thepipe:

$ mkfifo io$ { cat < io; echo cat received EOF; } &$ cat > io &$ cat > io &$ jobs[1]   Running                 { cat < io; echo cat received EOF; } &[2]-  Stopped                 cat > io &[3]+  Stopped                 cat > io &$ kill %3[3]+  Terminated              cat > io$ kill %2[2]+  Terminated              cat > iocat received EOF

When terminating all writers to io, the reader cat < io receivedEOF and terminated itself.)

So, when cat <o reads from the o pipe, the openssl process starts.It reads input from i while sending output to o. It quits once itsees EOF when trying to read again, before it receives the response fromthe remote site.

Now that we know that a reader sees EOF when the last writer quits, wecan open a file descriptor to the pipe. That file descriptor will stayopen until we close the terminal (or explicitly close it with exec 3>&-):

$ exec 3>i

This looks good:

CONNECTED(00000003)---Certificate chain 0 s:CN=www.bol.com   i:C=US, O=Let's Encrypt, CN=R3   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256   v:NotBefore: Mar 17 10:19:22 2024 GMT; NotAfter: Jun 15 10:19:21 2024 GMT... snip ...Verify return code: 0 (ok)---HTTP/1.1 301 Moved Permanentlydate: Mon, 01 Apr 2024 13:34:24 GMT... snip ...

This works beautifully. We can send requests from another terminal andcat <o keeps the output pipe open. If we want to interact with theoutput (e.g. grep it), however, need something else. We would like tojust keep the output pipe open, read whatever we want from the outputstream, play with it, make the next request, etc. We cannot simplyterminate cat <o, because it is the last reader from o. Whenopenssl writes to o while there are no readers left, openssl willterminate itself. That is because when a process tries to write to apipe that’s closed, it will receive the SIGPIPE signal from thekernel:

If all file descriptors referring to the read end of a pipe have beenclosed, then a write(2) will cause a SIGPIPE signal to be generatedfor the calling process.

And we did close the pipe by terminating cat <o. But openssl isnot writing anything, it just sits there, right? Right, but a TLSconnection is a two-way street and the remote server might sendsomething, like a session ticket renewal. openssl will try to writethis to the output pipe and terminate.

Opening another file descriptor that reads from the output should do it:

$ exec 3>i 4<o

This will keep both the input and output streams open its processends. Finally, we’re ready to have as many exchanges as we want:

$ h=bol.com$ openssl s_client -servername $h $h:443 <i >o &[1] 5414$ exec 3>i 4<o[2] 5415$ printf '%s\r\n' 'GET / HTTP/1.1' 'Host: bol.com' '' >i$ cat <o... snip certificate stuff ...HTTP/1.1 301 Moved Permanentlydate: Thu, 28 Mar 2024 09:14:54 GMT... snip ...^C$ printf '%s\r\n' 'GET / HTTP/1.1' 'Host: bol.com' '' >i$ cat <oHTTP/1.1 301 Moved Permanentlydate: Thu, 28 Mar 2024 09:15:24 GMT... snip ...^C

Upgrade Password Hashing Method for Existing Users

Mon, 11 Mar 2024 09:06:37 +0100

When you’ve been using the same Linux system for a while, you might wantto upgrade the password hashing methods used. This article explains howto discover the methods supported by your system, set the preferredmethod and then upgrade the password hashes to make use of the newhashing method.

You can see the supported hashing methods by looking at man 5 crypt. (Note that man section 5 indicates “Fileformats and conversions, e.g. /etc/passwd”.) Currently, onmy system, yescrypt is favored and recommended for new hashes.

To check your current hashing method on somebody’s account:

# getent shadow johnjohn:$6$JYJjfxe2YGsIGlvz$7bDibTmj/Iq.EelFEBJd24Ip66fVeBLGK/CMUyOl/9YpgIxtDfq2cLzzAc5b7sXhW3nCc2f4lKzMejFrBr3UF/:18133::::::

(You can read about the format of shadow by looking at man 5 shadow.) Again, looking at man 5 crypt, wesee that the $6$ in the second section after the first colon :indicates that this password was hashed with sha512crypt which,according to the same documentation is acceptable for new hashes. But weknow we can do better by using a “memory-hard” hash such as yescrypt.

passwd(1) changes user passwords. It will use thehashing method specified by the pam_unix.so module inthe PAM configuration (/etc/pam.d/passwd) or else look at theENCRYPT_METHOD variable in /etc/login.defs:

$ grep ^password /etc/pam.d/passwdpasswordincludesystem-auth$ grep ^password /etc/pam.d/system-authpassword   required pam_unix.so try_first_pass nullok shadowpassword   optional pam_permit.so

(Information about pam_unix.so arguments can be found in man 8 pam_unix.)

Here, we see that nothing explicit is said about which hashing methodshould be used for passwords. So we turn to/etc/login.defs:

$ grep ^ENCRYPT_METHOD /etc/login.defsENCRYPT_METHOD YESCRYPT

Now, when we upgrade our password with passwd, it will be protected bythe yescrypt ( $y$ ) hashing method:

# getent shadow johnjohn:$y$j9T$8Z3MNrSUYkSPZWcm83Dh$p.Qev5ogCpuF.KfGBjw1QfBBiV7m05PPWxIGuNtbJc9:19793:0:99999:7:::

You can create a yescrypt password hash yourself by running:

$ export PASS=pass SALT='$y$j9T$8Z3MNrSUYkSPZWcm83Dh$'$ perl -le 'print crypt($ENV{PASS}, $ENV{SALT})'$y$j9T$8Z3MNrSUYkSPZWcm83Dh$p.Qev5ogCpuF.KfGBjw1QfBBiV7m05PPWxIGuNtbJc9

You can read more about the format of the input tocrypt(3) in man 5 crypt under “FORMAT OFHASHED PASSPHRASES”. Suffice to say here that it consists of a prefix,options, a salt and a hash, separated by $. The prefix fieldindicates which has is to be used and changes what the other fieldsmean.

In this current example, I do not know the exact requirements for thesalt, for instance. I experimented and found that openssl rand -base64 15 produces a value thatcrypt(3) grokked.

(Incidentally, password verification is made easy, because you can justcompare the result of crypt(3) with the password hash asstored.)

The yescrypt hashing method takes a cost factor. This can be specifiedin /etc/login.defs under the YESCRYPT_COST_FACTORkey. Unlike the KDF rounds used with the sha256crypt and sha512cryptmethods, which increases complexity linearly, increasing the cost factorof yescrypt will lead to an exponential increase in memory usage andslow the password comparison down significantly. You might want toexperiment on your system and see what the highest setting is you stillfind bearable. Although you might not want to wait 5 seconds each timeyou log in or want to use sudo, keep in mind that it will takesomebody that cracks these the same amount of resources for each guess.

See yescrypt parameters (especially the Nparameter which is influenced by the YESCRYPT_COST_FACTOR key) and adescription of them on the Unix & Linux Stack Exchange.

Maven Wrapper Integrity Validation

Fri, 08 Mar 2024 14:46:48 +0100

Maven is modular. It uses plugins to achieve its goals. These pluginsare downloaded when they are invoked.

For example, executing mvn wrapper:wrapper will make Maven first lookin your local repository (~/.m2/repository), and, if it cannot findit, download the wrapper plugin from the repository defined in yoursettings.xml. (If no custom repository is defined, the default Maven“central” repository will be used.)

How does mvn know what to download, though, based only on a prefix(wrapper:)? Because the wrapper plugin used the Maven prefixrules. Maven looked in the current project, and not finding what itwas looking for downloadedhttps://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml.Now it found this:

<!-- ... snip... --><plugin>  <name>Apache Maven Wrapper Plugin</name>  <prefix>wrapper</prefix>  <artifactId>maven-wrapper-plugin</artifactId></plugin><!-- ... snip... -->

Maven now knows it can download this plugin from$REPO/org/apache/maven/plugins/maven-wrapper-plugin/$LATEST_VERSION/maven-wrapper-plugin-$LATEST_VERSION.jar.

Recently, the wrapper plugin added SHA-256 checksums to verify theintegrity of maven-wrapper.jar and the Maven distribution.

These SHA-256 sums are not easily discovered, however. I could not finda web source that listed them. Fortunately, signatures are uploadedtogether with the source code. The public signing keys can be foundelsewhere.

So, we first download all the Apache Maven certificates (“keys”) andsign them locally to make them valid:

$ curl -o apache-keys.txt https://downloads.apache.org/maven/KEYS$ gpg --import < apache-keys.txt$ while IFS=: read -r type validity key_len pubkey_algo key_id rest; do>   if [[ $type == pub ]] && ! [[ $validity =~ ^(e|r)$ ]]; then>       gpg --quick-lsign-key "$key_id">   fi> done < <(gpg --show-keys --with-colons apache-keys.txt 2>/dev/null)

(See for an explanation of the meaning of GnuPG’s colon-delimited fieldshttps://github.com/gpg/gnupg/blob/master/doc/DETAILS.)

We’ve now locally certified all the imported Apache Maven certificates.(This basically means you’ve attested that the UIDs in them belong tothe provided public keys. “Locally” just means they are not going to bepart of any export of the certificates, for example when uploading to akey server.) GPG has set the validity of these certificates to “full”.We now accept signatures on files and other data from thesecertificates.

We download the file manually and verify it using the detached signature(file ending in .asc for “ASCII-armored binary”):

$ gpg --verify maven-wrapper-3.2.0.jar.asc maven-wrapper-3.2.0.jargpg: Signature made Thu Mar  9 00:07:07 2023 CETgpg:                using RSA key 84789D24DF77A32433CE1F079EB80E92EB2135B1gpg:                issuer "sjaranowski@apache.org"gpg: Good signature from "Slawomir Jaranowski <sjaranowski@apache.org>" [full]gpg:                 aka "Slawomir Jaranowski <s.jaranowski@gmail.com>" [full]

Now that we have some confidence that we downloaded a legitimate copy ofthe maven-wrapper.jar file, we can calculate its SHA-256 sum.

The distributionSha256Sum should be the SHA-256 checksum of thedownloaded distribution, i.e. the actual Maven tool. According to thedocumentation, we can find the archives here:https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.6/apache-maven-3.9.6-bin.zip.Again, we download the file manually, let GPG verify it and calculatethe SHA-256 digest.

Now we’re ready to kick off the Maven wrapper plugin:

$ mvn wrapper:wrapper \>   -DwrapperSha256Sum=e63a53cfb9c4d291ebe3c2b0edacb7622bbc480326beaa5a0456e412f52f066a \>   -DdistributionSha256Sum=f00af914c785c9faed661f223000a92d1de9553f5c82d3b4362e66d9c031625f

This downloads the wrapper plugin which in its turn downloads thewrapper distribution (mvnw executable and .mvn directory containingthe wrapper configuration). The configuration file now looks like:

$ sed '/^#/d' .mvn/wrapper/maven-wrapper.propertiesdistributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.6/apache-maven-3.9.6-bin.zipdistributionSha256Sum=f00af914c785c9faed661f223000a92d1de9553f5c82d3b4362e66d9c031625fwrapperUrl=https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.2.0/maven-wrapper-3.2.0.jarwrapperSha256Sum=f00a53cfb9c4d291ebe3c2b0edacb7622bbc480326beaa5a0456e412f52f066a

What is protected by the checksums is not the execution of the mvnwshell script, but the execution of what that script executes:maven-wrapper.jar. When executed, this maven-wrapper.jar downloadsthe actual Maven distribution (mvn) if necessary and verifies thedownloaded file against the distributionSha256Sum in the config file.Now, if the wrapper SHA-256 sum is wrong, the shell script will throw anerror. And if the just-downloaded distribution’s SHA-256 sum is wrong,the maven-wrapper.jar will throw an error.

So, updating the Maven wrapper comes down to:

Download the maven-wrapper.jar and apache-maven-3.9.6-bin.zip
Download and verify their detached signatures
Calculate their SHA-256 digests
Execute mvn wrapper:wrapper -DwrapperSha256Sum=<digest> -DdistributionSha256Sum=<digest>

More information on the Maven wrapper plugin can be found here:https://maven.apache.org/wrapper/.

Prevent Exit When Receiving SIGPIPE with Pipefail Set

Thu, 29 Feb 2024 09:50:28 +0100

When a program that is part of a pipeline quits, it closes its stdin andstdout. Programs that produce the input for the quitted process may still tryto write to its stdin. Since they cannot, they will receive a SIGPIPE from thekernel.

You can see this in action:

$ seq 1 10000 | head -11$ echo "${PIPESTATUS[@]}"141 0

Conventionally, an exit status N greater than 128 indicates the programwas terminated by signal N - 128. Since SIGPIPE is signal 13, 141 - 128 = 13 indicates your program was ended by a SIGPIPE.

Source

When running scripts with set -o pipefail, this will terminate thescript, however. A workaround is to check for exit code 141 and exitthe pipeline with 0 instead:

seq 1 10000 | head -1 || \    { if [ "$(kill -l "$?")" = PIPE ]; then exit 0; else exit "$?"; fi; }

Create MacOS App Bundle from Script

Sun, 25 Feb 2024 09:23:10 +0100

I’m going to show how to create the smallest possible MacOS app bundle we can:one that simply executes a Bash script. Then, I’m going to expand a bit on thatand add an icon to it. I’ll also show how open a Terminal app window so we cansee the script’s output in it.

A MacOS app bundle is a directory that is interpreted in a special way byMacOS. It is an application name followed by the suffix .app (e.g.foo.app). A minimal app bundle would look like:

$ mkdir foo.app$ cat > foo.app/foo#!/bin/bash -izenity --calendar$ chmod u+x foo.app/foo$ tree foo.appfoo.app`-- foo

We create an app bundle foo.app that contains an executable of the samename: if you name your app bundle baz.app, the executable should be namedbaz. You can now execute the bundle by typing open foo.app in yourterminal, or double clicking on the bundle in Finder.

(Note that in order to display a Zenity calendar, I needed to make Bashinteractive. Leaving out the -i flag would not display anything. I haven’tfigured out yet why that is the case, but do send me a note if youknow.)

If you want more control, add an icon, and so on, we need to get moreelaborate.

Open a Terminal Window to Display the Output of a Shell Script

$ mkdir -p bar.app/Contents/{MacOS,Resources}$ cat > bar.app/Contents/MacOS/wrapper#!/bin/bashscript_path="$(dirname "$0")"/actual-scriptopen -a Terminal "$script_path"$ cat > bar.app/Contents/MacOS/actual-script#!/bin/bashecho do something useful$ chmod u+x bar.app/Contents/MacOS/{wrapper,actual-script}

Here, we created a wrapper script that will open the Terminal and execute theactual, useful script such that any output would be visible to the user.

(Important: make sure the names of the files and directories are spelledcorrectly. For example, it’s Contents (plural) and NOT Content. Failing todo this will make your life miserable because the app won’t work in Finder andwon’t display a useful error message. Or, when opening it from your terminal,it will spit out weird errors like “executable is missing”.)

Then, add <name>.app/Contents/Info.plist:

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict>  <key>CFBundleName</key>  <string>Bar</string>  <key>CFBundleExecutable</key>  <string>wrapper</string>  <key>CFBundleIdentifier</key>  <string>com.relentlesscoding.Bar</string>  <key>CFBundleVersion</key>  <string>1.2.3</string></dict></plist>

CFBundleExecutable contains the name of the executable located in<name>.app/Contents/MacOS. Reminder: do not forget to chmod u+x <executable>.

CFBundleIdentifier should contain a unique identifier for your app. Fail tomake this unique, and your app will replace some other app in your Launcher.

For good measure, I added CFBundleVersion and put the app version in there.

Incidentally, you can read all about these properties in Apple’sdocumentation.

Add an Icon

If you search online, you’ll find recommendations for application icons andsizes. If you want to keep things simple, however, just download a 512x512 PNGicon from the internet, name it whatever you want, and put it in<name>.app/Contents/Resources.

Then, modify <name>.app/Contents/Info.plist to point to the icon. Here, Iadded an icon bar.app/Contents/Resources/bar.png:

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict>  <!-- ... snipped other tags... -->  <key>CFBundleIconFile</key>  <string>bar.png</string></dict></plist>

Tips

You can check the validity of the Info.plist file by running plutil path/to/Info.plist:
```
$ plutil bar.app/Contents/Info.plistbar.app/Contents/Info.plist: OK
```
MacOS won’t immediately pick up changes you make to the app bundle. You canforce its hand by touching the bundle:
```
$ touch bar.app/
```
All of this was tested on MacOS Sanoma.

Make passmenu Show GUI Pinentry When Using Pinentry Curses by Default

Sun, 11 Oct 2020 11:28:23 +0200

Let’s take a look at how to use a GUI pinentry for one program, and cursesfor everything else.

I’m on Arch Linux using i3wm as a window manager and pass as my passwordmanager. pass comes with a handy script, passmenu, that allows you to findyour password from within the status bar, dmenu. This way you won’t have toopen a terminal every time you want to copy a password.

However, I had instructed gpg-agent to use /usr/bin/pinentry-curses. Thismeant that every time I invoked passmenu and my PGP password was not alreadycached, nothing appeared to happen. (In fact, it would open thepinentry-curses in a terminal not accessible to me and time out.)

So here is how to use a different pinentry for passmenu and pass.

First, create a wrapper script for pinentry (~/bin/pinentry-wrapper):

#!/usr/bin/env bash## Defaults to Qt, with a choice of curses for selected programs# PINENTRY_USER_DATA is a GnuPG defined variable (see man gpg)case "$PINENTRY_USER_DATA" in    gtk)        exec /usr/bin/pinentry "$@"        ;;    *)        exec /usr/bin/pinentry-curses "$@"        ;esac

Make this script executable:
```
$ chmod u+x ~/bin/pinentry-wrapper
```
Instruct GnuGP to use your version of pinentry (~/.gnupg/gpg-agent.conf):
```
pinentry-program /home/neftas/bin/pinentry-wrapper
```
Restart the gpg-agent:
```
$ pkill -HUP gpg-agent
```

Create a wrapper script for passmenu (~/bin/passmenu):

#!/usr/bin/env bashPINENTRY_USER_DATA=gtk /usr/bin/passmenu "$@"

Make it executable:
```
$ chmod u+x ~/bin/passmenu
```
Make sure ~/bin is searched first in your PATH (put this in your .bashrc):
```
$ export PATH="~/bin:$PATH"
```

Check your work:

$ command -v passmenu/home/neftas/bin/passmenu

I’ve also posted this as an answer to a related question on the Unix & LinuxStackExchange page.

Why PINENTRY_USER_DATA and not MY_OWN_VAR? man pgp tells us that:

PINENTRY_USER_DATA    This value is passed via gpg-agent to pinentry. It is useful to convey extra    information to a custom pinentry.

Readline .inputrc Configuration Examples

Wed, 02 Sep 2020 19:10:49 +0200

These are some of my favorite readline tweaks.

To make changes to readline’s behavior, you have to put configuration in a filein your home directory called .inputrc. To include commands from/etc/inputrc, put the following first:

$include /etc/inputrc

Variables to customize readline behavior can be set in this file by statementsof the form:

set variable-name value

Restart your terminal emulator for the changes to take effect, or execute incurrent window:

$ bind -f ~/.inputrc

Briefly Move to Opening Parenthesis When Closing Parenthesis Is Typed

You want to make sure you have an even amount of opening and closingparentheses in your statements. By setting

set blink-matching-paren On

the cursor will briefly move to the opening parenthesis.

Display Possible Completions With Different Colors to Indicate File Type

Say, you’re looking for file starting with .X, so you type:

$ ls .X

and then press Tab. Setting

set colored-stats On

will display the possible completions with a different color:

Display Completions One Per Line

Instead of showing completions in columns, you can display completions one perline by setting:

set completion-display-width 0

The default is -1, which means the value is ignored and completions areshown in columns.

Ignore Case on Completion

You want to cd into the folder Documents, so you type:

$ cd docu

and press Tab. Nothing happens. You want readline to complete itfor you ignoring the case:

set completion-ignore-case On

Treat Hyphens and Underscores as Equivalent

If you enabled completion-ignore-case (see above), you may also wantreadline to treat hyphens - and underscores _ as equivalent whenperforming completion:

set completion-map-case On

Replace Common Prefixes of Certain Length with Ellipsis

Instead of seeing the beginning of file names that you have already typed, youwant to focus on what differentiates the files:

$ lsfile1 file2 file3 file4$ ls file<TAB>...1...2...3...4

When you type ls file followed by Tab, you will see the commonprefixes replaces by three dots.

Disable the “Display all 768 possibilities? (y or no)” on Completion

Sometimes, when the amount of possible completions is greater than 100(default), readline will show a question whether or not you want to see allpossibilities. If you want to increase the threshold (default is 100), you canset:

set completion-query-items 1000

Turn Current Command Into a Comment

You’re typing a command and halfway realize you need to do something elsefirst. But you don’t want to throw away what you typed away. Then you can typeM + # in emacs mode or # in vi command mode(in vi insert mode you can still type M + # for the sameeffect).

You can change the string that is inserted when you press the insert-commentkey combo with set comment-begin #.

Safe Pasting with Bracketed Paste

If you copy commands from the web, you might have noticed that when you copy anewline, Bash will interpret that newline as an Enter and execute whatever itis in your input buffer at the moment. This is unsafe, because what you seemight not be what youcopy. You can be avoidedby setting:

set enable-bracketed-paste On

This command puts an escape sequence around what you copy and paste, so Bashcan tell the difference between what you, the user types and what you copiedfrom somewhere.

The default is Off.

Expand Tilde to Home Directory

When you refer to the home directory of the user you’re currently logged inas, you most likely use tilde ~ as in cp ~/thesis.txt ~/Documents/.Setting

set expand-tilde On

will always expand the tilde to the full path of your home directory when wordcompletion is attempted (meaning you need to press Tab at somepoint).

The default is Off.

Sudo With Examples

Sat, 15 Aug 2020 16:03:39 +0200

I’ve recently finished “Sudo Mastery” by Michael W. Lucas.He’s a great and fun author that writes both technical guides and novels. Ibought his "$ git commit murder" some years agoand was sold.

This post is basically a write-down of what I learned about sudo from hisbook.

Why Use `sudo`?

The most common reason to use sudo is to not hand out the root password toevery person that logs on to a server. The problem with handing everyone theroot password is that:

They would be able to arbitrarily execute every command as root (there’s nolimiting the commands that users can run as root).
There would be no audit trail other than that root executed these commands.
If you want to revoke someone’s root access, you would have to change the rootpassword and communicate the new password to the other root users.

sudo aims to solve these problems:

You can specify which users can execute which commands (even includingparameters to commands).
sudo can log all commands that are executed and notify when a user fails toauthenticate, tries to execute commands she is not allowed to and even whenshe tries but is not allowed to run sudo at all.
To revoke sudo access, the sysadmin just removes the user from the sudoerspolicy.

Now, when you are the sole sysadmin of a server, sudo might be overkill foryou. You can just use su to become root and execute privileged commands.However, running all commands as root might be dangerous if you make mistakes.Having to type sudo !! (“this time I mean it, dammit!”) might give you alittle time to think the command you’re about to execute over.

What Everybody Knows About `sudo`

Run command as root:

$ sudo docker ps

Run command as a different user:

$ sudo -u john docker ps

Run command as a different group:

$ sudo -g http less /var/log/nginx/access.log

Become root by using your own password:

sudo -i

The -i (long: --login) makes sure you get an environment that is equal towhat you would get if you logged in as root. This means the environment fromthe user running sudo is not kept, and root’s login-specific resource files(.profile, .bash_profile, etc.) will be run. Specifying -i/--login isrecommended.

What Does a Simple sudo Policy Look Like?

Sudo stores its policy in /etc/sudoers. It contains rules, defaults andaliases for hosts, users, run-as groups and commands.

Let’s take a look at a rule.

stefan  www = (root:admin) NOPASSWD: /usr/bin/docker

This policy specifies that the user stefan can execute the command/usr/bin/docker as user root or group admin on the machine with hostnamewww and does not need to provide a password.

The syntax of a rule in sudoers looks like:

<user> <host> = (<user>:<group>) <tag> <commands>

A rule could be as simple as:

stefan ALL = ALL

This gives user stefan carte blanche to execute whatever command tickles hisfancy.

Instead of a Single User, Specify a List of Users

stefan,tom,rik www = /usr/bin/docker

Or you could define a User_Alias and use that:

User_Alias ADMIN = stefan,tom,rikADMIN www = /usr/bin/docker

To permit all users to execute a command as root, use the catch-all ALL.

Instead of a Single Machine, Specify a List of Machines

stefan www1,www2,www3 = /usr/bin/docker

Or you could define a Host_Alias and use that:

User_Alias DBAS = henk,trudy,janeHost_Alias DBS = pg-db-tst,pg-db-acc,pg-db-proDBAS DBS = /usr/bin/psql

This allows all DBAs (users henk, trudy and jane) to execute/usr/bin/psql on the machines pg-db-tst, pg-db-acc and pg-db-pro.

Just as with users, you could specify ALL as a catch-all to specify allmachines.

Instead of a Single Run-As User or Group, Specify a List of Users or Groups

The run-as specification of a rule (the optional (user:group) part),likewise, can be a list of users and groups:

stefan ALL = (root,mike,jane:root,admin,log) ALL

We could use a Runas_Alias:

Runas_Alias = root,mike,jane,%root,%admin,%log

To specify a group in a Runas_Alias, prefix it with %.

Again, (ALL:ALL) could be used to indicate that the rules allows executinga command masquerading as anybody or as a member of any group. Use (ALL)without the :<group> to allow becoming just any user.

Use Tags to Change `sudo` with Regard to Behavior of Commands

NOPASSWD: is called a “tag” and you can provide several of them on a singlerule:

stefan  www = (root:admin) SETENV: NOPASSWD: /usr/bin/docker

These tags change some aspect of the commands following it. In our example,SETENV: allows the user to use the -E flag (preventing its environmentvariables from being stripped by sudo) and to keep environment variables set onthe command line:

sudo DOCKER_UID=$(id -u http) /usr/bin/docker

Normally, these variables would be stripped (see man sudoers for moredetails):

$ sudo FOO=bar printenv FOOsudo: sorry, you are not allowed to set the following environment variables: FOO

The NOPASSWD: tag allows the user to run the command without entering apassword.

Provide a List of Commands in a Single Rule or Use Command Aliases

stefan www = (root) /usr/bin/docker,/usr/bin/docker-compose

Alternatively:

Cmnd_Alias DOCKER = /usr/bin/docker,/usr/bin/docker-composestefan www = (root) DOCKER

Ground Rules

Use `visudo` to Edit the Sudoers Policy

The policy sudo uses is stored in /etc/sudoers. If you syntactically break thepolicy, sudo won’t run. Therefore, always edit it using visudo. visudocopies the sudoers file to a temporary file and allows you to edit it using yourfavorite editor. When you close the temporary file, it validates the syntaxbefore overwriting /etc/sudoers. If the syntax is invalid, it will show aprompt with the line number that contains the error and asks you what to do.This allows you to edit the file again (press e andEnter), fix the mistake and save the correct version.

WARNING

Before starting to experiment with sudo, make sure you can su to root soyou can fix your sudoers policy. It will be a nasty surprise when you screw upthe sudoers policy and only then find out you cannot become root through su.

Whitelist, Not Blacklist

As is common with setting up firewalls, use a whitelist approach rather than ablacklist approach. It is hard to enumerate all the programs a user is notallowed to execute. Rather, explicitly specify the programs a user needs to doher job.

stefan ALL = /usr/bin/createdb,/usr/bin/dropdb,/usr/bin/psql

Rather than

stefan ALL = !/usr/bin/passwd,!/usr/bin/mount,!/usr/bin/hostnamectl

Set `noexec` to Prevent Shell Escapes

If you allow users to execute editors and pagers as root, you basically givethem the keys to the kingdom. Why? Editors such as vi and vim, and pagerssuch as less and more allow users to execute commands in a subshell. Try ityourself: open a file with less, type an exclamation mark ! and then ls.It will list the files in the directory you executes less from. These commandsare executed as the user that runs less, which is root if we had used sudo less file.txt. If, instead of ls, we had typed bash, we would have a rootshell.

To prevent this from happening, you can set the noexec parameter, either as adefault for everyone (recommended) or per-command:

Defaults!ALL noexec# alternativelystefan ALL = ALL, NOEXEC: /usr/bin/less

What this does is prevent executables from running further commands itself.After setting this, executing ls from within less now returns:

!done  (press RETURN)

However, you may find that by setting a global noexec parameter you also can’trun visudo anymore:

$ sudo visudovisudo: unable to run /usr/bin/vim: Permission deniedvisudo: /etc/sudoers.tmp unchanged

Oops. visudo tries to spawn a text editor, but it cannot run additionalcommands and so it fails. We need to exclude visudo from the default rule:

Defaults!ALL noexecDefaults!/usr/bin/visudo !noexec

However, we might not want to give everybody access to visudo. A good way tosolve this is with a command alias:

Defaults!ALL noexecCmnd_Alias MAYEXEC = /usr/bin/visudostefan ALL = ALL, EXEC: MAYEXEC

Last Rule Matches, Wins

What does the following sudoers policy do:

stefan ALL = !/usr/bin/docker,!/usr/bin/docker-composestefan ALL = ALL

You’ve guessed it: it allows user stefan to execute any command it wants. Thislooks straightforward, but don’t forget that a sudoers policy might containincludes. For example, the sudoers policy shipped with Arch Linux at themoment of this writing contains this at the end:

#includedir /etc/sudoers.d

Sudo will read every file in the specified directory in lexical order. Thismeans you need to understand what lexical ordering entails, or start every filewith a zero-padded number (001-, 002-, etc.).

Every file could have rules that supersede the rules in sudoers.

Use `sudoedit`, Not `sudo ed`

Because editing a file as another user is such a common task, a conveniencecommand sudoedit was created. This creates a temporary copy of the file thatcan be edited by the user invoking sudo (no elevated privileges needed) andwhen the file gets closed, it replaces the old file with the edited copy. Thebenefit of using sudoedit is that a user cannot abuse his elevated privilegeswith shell escapes, because the editor is run as an ordinary (non-privileged)user.

Common Tasks in sudo

Change Default sudo Editor

Defaults editor = /usr/bin/ed

You can specify more than one by separating them with colons:

Defaults editor = /usr/bin/ed:/usr/bin/ex:/usr/bin/vi:/usr/bin/vim

This preferences is used for both visudo and sudoedit, unless one of theSUDO_EDITOR, VISUAL or EDITOR environment variables are set. Keep in mindthat sudo strips these environment variables unless explicitly allowed in theenv_keep list.

Extend Authentication Timeout Period

By default, sudo will ask you to reauthenticate every five minutes. If you usesudo a lot, you might want to increase this interval.

Defaults timestamp_timeout = 30  # minutes

Authenticate in One Terminal, Be Authenticated Everywhere

By default, sudo will require you to authenticate once per terminal. So, ifyou open two terminals, authenticate in the first, and then run sudo in theother, you will be required to authenticate again. You can change this behaviorby setting timestamp_type to global (authenticate once and you’reauthenticated in all your user’s sessions on the same machine), ppid (commandsthat run from the same parent process, say a shell, are all authenticated) ortty (user’s sessions are authenticated separately per terminal, which is thedefault).

Defaults timestamp_type = global

Passwordless sudo For Some Commands

To run some commands with elevated privileges without having to enter apassword:

stefan ALL = NOPASSWD: /usr/bin/docker

Keep Select Environment Variables

By default, sudo removes most of the environment variables from the invokinguser’s environment, because they change the behavior of commands andinappropriate environment variables may break a system. To explicitly allowcertain variables:

Defaults env_keep += "EDITOR SYSTEMD_EDITOR"

Note the double quotes around the environment variables. Also note that we’reappending to the env_keep list. Use = instead of += to replace the list.

Run sudo -V as root (sudo sudo -V) to see which variables get stripped byyour installation of sudo and which environment variables are already inenv_keep.

Flush Authentication Timestamp

When stepping away from your computer, you might want to flush your sudocredentials, so that nobody will be able to run sudo with your cachedcredentials. Run sudo -K (long: --remove-timestamp) to remove your cachedcredentials everywhere from the system.

Intruder Detection by Using Command Digests

sudo provides functionality to check if the command you want to run has beentampered with. You can provide a digest of the command and sudo will check itbefore running the command.

Generate a sha224 digest of /usr/bin/docker:

$ openssl sha224 -binary /usr/bin/docker | base64EQy+FoFpPakt/QdMqJGJQLLeWMHmsFWG0N1A8Q==

Put the name of the selected digest, a colon and the digest before the command:

stefan ALL = sha224:EQy+FoFpPakt/QdMqJGJQLLeWMHmsFWG0N1A8Q== /usr/bin/docker

When you update the packages on your system, the digest will change. If you planto use digests, you will probably want to write a script that updates thedigests after every upgrade.

Replay sudo Sessions with `sudoreplay`

To see what your users have been up to, you can replay entire sudo sessionskeypress-by-keypress with sudoreplay:

Defaults log_outputDefaults!/usr/bin/sudoreplay !log_outputCmnd_Alias      REBOOT = /sbin/halt, /sbin/reboot, /sbin/poweroffDefaults!REBOOT !log_output

log_output tells sudo to log all output send to the terminal (if you changethis to log_input it will also log everything the user types). The next linestell sudo not to log the output of sudoreplay itself and not to log areboots and shutdowns. Trying to log a reboot or shutdown can delay the process,because sudo might try to write logs to a disk that has just been unmounted aspart of it.

sudoreplay offers extensive functionality to find the session you’reinterested in. However, that is outside the scope of this post. Read man sudoreplay for more information.

Read man sudo (sudo command), man sudoers (sudo policy) or if you’re lazy,buy Michael W. Lucas’ book!

Read File into JSON List with jq

Wed, 22 Jul 2020 07:48:14 +0200

You have a file and you want to convert the contents into a JSON list. How canwe leverage jq to do that?

$ cat fileonetwothree$ jq --null-input --raw-input '[inputs]'[  "one",  "two",  "three"]

--raw-input (short -R) takes each line in the input as a string and passesit to the filter, instead of interpreting the file as containing JSON.

--null-input (short -n) is important:

$ printf '%s\n' one two three | jq --raw-input '[inputs]'[  "two",  "three"]

--null-input doesn’t read the input at all. Instead, it will run the filterwith null as input. In our case, jq encounters inputs here, which triggersan I/O operation that reads the contents of the file (or files) as strings(because of --raw-input) into a list (because inputs is surrounded by[...]).

Without --null-input, jq takes the first line of the file, and passes thatto the filter. The filter discards this input (we don’t use the identityoperator .) and proceeds to read all remaining inputs (two and three).Obviously, this is not what we want.

Knowing this, we could do the following (but really shouldn’t):

$ printf '%s\n' one two three | jq --raw-input '[.] + [inputs]'[  "one",  "two",  "three"]

Remove Empty Strings from Output

If you get empty strings for whatever reason in your output, you can removethem:

$ printf '%s\n' '' one two three | jq -n -R '[inputs]'[  "",  "one",  "two",  "three"]$ printf '%s\n' '' one two three | jq -n -R '[inputs | select(length > 0)]'[  "one",  "two",  "three"]

Pipe to Multiple Commands with Bash

Mon, 13 Jul 2020 18:27:25 +0200

Sometimes it’s handy to filter the output of a command (with grep, forexample) while still having the column names (the first line) available. How dowe go about that?

One example of this would be when filtering the output of ps. What does thefollowing output mean, after all?

$ ps -ef | grep logd    0   124     1   0  7:45AM ??         0:01.46 /usr/sbin/syslogd    0   146     1   0  7:45AM ??         0:09.44 /usr/libexec/logd 1000 17263 14631   0 10:36AM ttys010    0:00.01 grep --color=auto logd

It would be much more readable if the output would be:

 UID   PID  PPID   C STIME   TTY           TIME CMD   0   124     1   0  7:45AM ??         0:01.46 /usr/sbin/syslogd   0   146     1   0  7:45AM ??         0:09.44 /usr/libexec/logd1000 17263 14631   0 10:36AM ttys010    0:00.01 grep --color=auto logd

Now, the correct answer to this is:

$ ps -ef | tee >(sed -n 1p) >(grep logd) >/dev/null

Here, tee takes a variable number of files to write the same output to. We useprocess substitution (see man bash) to make tee write to the named pipes,which are a kind of file. tee also prints its input to standard out, and wesuppress that by redirecting it to /dev/null.

Because the list of commands inside process substitution run asynchronously, toensure sed -n 1p runs before grep logd, we could insert sleep 0.1s beforerunning grep logd. That should give sed -n 1p enough time to print its lineand will make the output more reliable.

Interestingly, >(head -1) instead of >(sed -n 1p) would not work underLinux, because head closes its stdin as soon as it has read enough bytes toprint the first line. At that point, tee quits with an exit status of 141 (128+ 13 for SIGPIPE – see man bash and grep for “Simple Commands” for anexplanation). Instead, we use a command which is guaranteed to read the entireoutput tee feeds it.

In the rest of this post I will look at another “solution” to this problem thatuses group commands, and show why it is incorrect.

The Wrong Answer

I read the comments below the answer on ServerFault on “How to grep ps outputwith headers” which suggested wecould use ps -ef | { head -1; grep logd; } to solve this problem(incidentally, { cmd; cmd; } is called a “group command”). If we wouldn’tknow from our previous run what the correct output would look like, this wouldseem to work:

$ ps -ef | { head -1; grep logd; }  UID   PID  PPID   C STIME   TTY           TIME CMD    0   146     1   0  7:45AM ??         0:09.48 /usr/libexec/logd

But notice two of the three processes are missing. What gives?

The key is understanding that cmd1 | { cmd2; cmd3; } does not provide a copyof the standard output of cmd1 to cmd2 and cmd3 (so that both cmd2 andcmd3 would be working with the same input). Rather, the input is shared:whatever is consumed by cmd2 is no longer available for cmd3.

We can prove this very easily. If we substitute cmd2 with something thatconsumes all input, such as grep (which has to consume every line in the inputlooking for a match), the input for the next command should be empty:

$ printf '%s\n' foo bar baz | { grep nomatch; wc -c; }0

Indeed, we see that wc -c has zero bytes to work with.

OK, you might think, but head -1 only consumes the first line (up until thefirst newline) of the input, right? So why exactly doesn’t ps -ef | { head -1; grep logd; } work? As it turns out, head reads more than just up to thefirst newline:

$ printf '%s\n' foo bar baz | { head -1; wc -c; }0

How much does it read by default (I’ve tested with head from GNU coreutils 8.32)?

$ < /dev/urandom tr -d -c '[[:alnum:]]' | head -c 10000 > urandom.txt$ cat urandom.txt | { head -1 >/dev/null; wc -c; }8976$ echo '10000 - 8976' | bc1024

It reads the first 1024 bytes that are no longer available to any othercommands that are following. This means that you shouldn’t rely on groupcommands to work with a single input.

Maven's Versions Plugin Updates All Your Dependencies With a Single Command

Mon, 11 May 2020 19:25:43 +0200

It’s a good habit to use the latest versions of the dependencies youuse. Not only because the updated version might contain new features orbetter performance (one can hope), but also because bugs are fixed andsecurity issues are tackled. If you go about it by hand, however,updating dependencies is a boring and tedious task. Luckily for us,there is the Maven versions plugin to help us.

Check Which Dependencies/Plugins/Properties Need Updating

If you need a report on which dependencies, plugins or properties usedfor versioning can be upgraded, run one of these:

versions:display-dependency-updates scans a project’s dependenciesand produces a report of those dependencies which have newer versionsavailable.

versions:display-plugin-updates scans a project’s plugins andproduces a report of those plugins which have newer versionsavailable, taking care of Maven version prerequisites.

versions:display-property-updates scans a project and produces areport of those properties which are used to control artifact versionsand which properties have newer versions available.

Update Parent

If you have a parent section, the following command updates the versionto the latest available:

$ mvn versions:update-parent

Update To Latest Versions

To upgrade all your dependencies to the latest versions, use:

$ mvn versions:use-latest-versions

Update a Particular Property with Bounds

To upgrade dependencies that get their version from the <properties>section of your POM to version that has bounds:

$ mvn versions:update-property \    -Dproperty='cucumber.version' \    -DnewVersion='(,4.99]'

In this case, the lower bound is unspecified (and exclusive asindicated by the () and the upper bound in 4.99 inclusive. You cantweak -DnewVersion to yourliking.

Conclusion

The Maven versions plugin makes it easy for us the see whichdependencies are outdated and can even update the dependencies for uswith a single command. After updating, don’t forget to run your testsuite and run the application to make sure nothing broke!

Reference

This has been tested on MacOS with the following Maven and versionsversions:

$ mvn --versionApache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)Maven home: /usr/local/Cellar/maven/3.6.3_1/libexecJava version: 13.0.2, vendor: N/A, runtime: /usr/local/Cellar/openjdk/13.0.2+8_2/libexec/openjdk.jdk/Contents/HomeDefault locale: en_NL, platform encoding: UTF-8OS name: "mac os x", version: "10.15.4", arch: "x86_64", family: "mac"$ mvn help:describe -Dplugin=versions -Dminimal | grep '^Version: 'Version: 2.7

The documentation can be foundhere.

Oracle Sqlplus in a Small Docker Container

Mon, 11 May 2020 19:19:22 +0200

When you only need Sql*Plus, it’s good to have a small Docker image that youcan use. This post will look at how to create this Docker image.

Introduction

Some time back I wrote a blog post about SQL*Pluscommands. To run those commands, I woulduse a 15 GiB image of Oracle XE (Express Edition) just to be able to use acommand-line tool. Today I stumbled upon Oracle’s Docker images, that are awhole lot smaller.

How to Build Your Own Image

Oracle provides Dockerfiles on its GitHubpage. I’m interested in justSQL*Plus so I can change my password on an Oracle database and maybeexecute some simple queries. I took a look at thisDockerfile,and modified it slightly to include only what I need:

FROM oraclelinux:7-slimARG release=19ARG update=6RUN  yum -y install oracle-release-el7 && \     yum-config-manager --enable ol7_oracle_instantclient && \     yum -y install oracle-instantclient${release}.${update}-basic \        oracle-instantclient${release}.${update}-sqlplus && \     rm -rf /var/cache/yumCMD ["sqlplus", "-v"]

Build the image and tag it as sqlplus:

$ docker build -t sqlplus .

The image it generates is pretty small, weighing around 370 MB. (OK, animage based on Alpine should be smaller, but Oracle requires glibc andnot musl libc.) You can create a container and run a SQL*Plus commandwith:

$ docker run -it sqlplus user/password@db.domain.tld:1521/FOO

Optional: Add `tnsnames.ora` to the Image

In Oracle, a file name tnsnames.ora contains the connection stringsfor databases. They allow you to connect to a database using sqlpluswithout specifying the hostname:

$ sqlplus user/passsword@FOO

Add the tnsnames.ora to the image by putting the following between theRUN and CMD directive in the Dockerfile:

COPY ./tnsnames.ora /rootENV TNS_ADMIN=/root

Automatic Rollback of Transactions in Spring Tests

Sun, 01 Mar 2020 11:11:02 +0100

Put@Transactionalon your test class or methods and test-managed transactions willautomatically be rollbacked.

The Antipattern

When looking at system test in Spring, it is not uncommon to see thispattern:

import org.junit.jupiter.api.Assertions.assertEqualsimport org.junit.jupiter.api.extension.ExtendWithimport org.junit.jupiter.api.Testimport org.springframework.beans.factory.annotation.Autowiredimport org.springframework.boot.test.context.SpringBootTestimport org.springframework.test.context.junit.jupiter.SpringExtension@ExtendWith(SpringExtension::class)@SpringBootTestclass TransactionalTestApplicationTests {    @Autowired lateinit var mapper: Mapper    @Test    fun `insert user into db, verify and do cleanup`() {        val user = mapper.createUser(username = "foo")        assertEquals("foo", user.username)        mapper.deleteUser(username = "foo")    }}

The cleanup is done manually, at the end of the test. This introduces aproblem: what if the test fails? Now, the cleanup statement at the endis no longer executed. A common solution is to introduce an @Aftermethod that does the cleanup in a more or less crude manner:

@Afterfun cleanup() {    mapper.deleteAllUsers()}

This pattern is in fact unnecessary. Spring provides the@Transactional annotation that will do an automatic cleanup afterevery @Test method:

@ExtendWith(SpringExtension::class)@SpringBootTest@Transactionalclass TransactionalTestApplicationTests {    @Autowired lateinit var mapper: Mapper    @Test    fun `insert user into db, verify and do cleanup`() {        /* code */    }}

Difference Between Test-, Spring- and Application-Managed Transactions

A test-managed transaction is one that is managed byTransactionalTestExecutionListener (or programmatically byTestTransaction).A test-managed transaction differs from Spring-managed transactions(transactions directly managed by Spring within the ApplicationContextthat is loaded for the test) and application-managed transactions(transactions programmatically managed within the application code thatis executed by the test):

Spring-managed and application-managed transactions will typicallyparticipate in test-managed transactions; however, caution should betaken if Spring-managed or application-managed transactions areconfigured with any propagation type other thanREQUIREDorSUPPORTS.— Documentation forTransactionalTestExecutionListener

What Does `@Transactional` Do?

Running tests while starting up the Spring application context, aTransactionalTestExecutionListeneris automatically configured. This listener provides support forexecuting tests with test-managed transactions. It notices the@Transactional on the test class or individual @Test methods andcreates a new transaction that is then automatically rolled back aftertest completion. The default is rollback. This behavior can be changed,however, by putting@Commitor@Rollbackat the class or method level. If you need more control over yourtransactions, you can use the static methods inTestTransaction.They allow you to start and end transactions, flag them for commit orrollback or check the transaction status.

Beware: Only Test-Managed Transactions Are Rolled Back

Transactions are not rolled back when the code that is invoked does notinteract with the database. An example would be writing an end-to-endtest that uses RestTemplate to make an HTTP request to some endpointthat then makes a modification in the database. Since the RestTemplatedoes not interact with the database (it just creates and sends an HTTPrequest), @Transactional will not rollback anything that is an effectof that HTTP request. This is because a separate transaction, one notcontrolled by the test, will be started.

Set Up or Tear Down Outside of a Transaction

You can annotate a public void method with@BeforeTransactionor@AfterTransaction.This indicates to Spring that this method should be run before or aftera test method is run within a transaction.

Spring Boot Version

I wrote this code with Spring Boot version 2.2.5.

SQL Injection in Bash Scripts

Sat, 29 Feb 2020 16:36:49 +0100

Problem

When working on making a small application with a ZenityGUI,I ask the user for input. I need to somehow sanitize this input, or Iwill leave the gates wide open for two problems:

In this blog post, I’m going to look at a solution for the firstproblem.

Disclaimer

Of course, you shouldn’t write SQL queries in a programming languagethat doesn’t provide a way to supply parameterized queries. Also, when auser is executing a shell script themselves on their local machine, SQLand command injection become academic, because the user already has theright to modify the database or to execute random Bash commands. Itwould only be problematic if the script would be executed by an HTTPserver that can reached by the internet at large. But in that case, youwouldn’t be writing a Bash script, you would use a proper programminglanguage, right?

Solving SQL Injection The Normal Way: Parameterized Queries

The solution for SQL injection, is, of course, parameterized queries.You send a query to the database and leave placeholders (AKA “bindvariables”) open, that take the form of a question mark or a namedparameter:

SELECT  *FROM    userWHERE   name = ?AND     password = ?;

SELECT  *FROM    userWHERE   name = :nameAND     password = :password;

The parameters are sent separately to the RDBMS, and are bound to acertain type (varchar, integer, etc) and thus cannot cause unintendedside effects anymore.

However, Bash Doesn’t Let You Do This

In Bash, however, it’s hard to use this if the provided executabledoesn’t accept parameters. Postgres’ psql takes a -v option, andallows you to write:

$ psql -u user -h host -v 'name=qwerty' -v 'password=12345' \    'SELECT * FROM user WHERE name = :name AND password = :password;'

MariaDB/MySQL, however, provides nothing of the kind.

IFS='|' read -r description amount < <(zenity --forms \        --add-entry description \        --add-entry amount \        --text 'Please specify a description and amount')# assume env variable MYSQL_PWD is setmysql -u user -h host \    -e "INSERT INTO advance VALUES ('$description', $amount);"

The code fragment above leaves us completely exposed to SQL injection.

Solving SQL Injection In Bash

We could base64 encode the userdata and use a function on the RDBMS to decode from base64.

IFS='|' read -r us_description us_amount < <(zenity --forms \        --add-entry Description \        --add-entry Amount \        --text 'Please specify a description and amount')description_base64="$(echo -n "$us_description" | base64)"amount=$(parse_float "$us_amount") || { >&2 echo 'Invalid float'; exit 1; }# assume MYSQL_PWD is setmysql -u user -h host \    -e "INSERT INTO advance VALUES (FROM_BASE64('$description_base64'), $amount));"

Why Does This Work?

You might think to yourself: why does this actually work? Doesn’t thefunction get expanded and you end up with the same problem? Let’s figureout how a SQL statement gets processed by the RDBMS.

When the RDBMS receives the SQL statement, it is handled by a SQL interpreter.The interpreter consists of two parts: a parser and an optimizer. First, theparser checks for syntax errors. If the syntax is correct, it will checkwhether the statement is semantically correct: for example, do all referencedtables and columns exist? The parser separates the pieces of a SQL statementinto a data structure that other routines can process. (As we’ll see, this isimportant: after parsing, SQL injection is no longer possible.) After theinterpreter is done with the statement, the statement is handed to theoptimizer. The optimizer tries to determine how to get the best performance.The result is an execution plan, which is fed to a code generator and finallyrun by the database processor.

Schematically:

(Readherehow Oracle processes statements). A function is evaluated during theexecution phase. The parser has no problem with a valid, existingfunction that promises to return a value that is expected in thatposition. The optimizer’s task is only to determine at what moment inthe execution phase the function is best executed, but also doesn’tevaluate the function. We can prove all this to ourselves by calling afunction that will throw an error:

MariaDB [foo]> \WShow warnings enabled.MariaDB [huishouden]> SELECT FROM_BASE64('!');+------------------+| FROM_BASE64('!') |+------------------+| NULL             |+------------------+1 row in set, 1 warning (0.004 sec)Warning (Code 1958): Bad base64 data as position 0MariaDB [foo]> -- no warning will show because function is not evaluatedMariaDB [foo]> SELECT FROM_BASE64('!') WHERE 1 = 0;Empty set (0.003 sec)

Calling FROM_BASE64 with an invalid base64 string ! results in aruntime exception. When we add a predicate in the WHERE clause thatalways fails (1 = 0), the query gets executed nonetheless, showing usthat MariaDB/MySQL will only evaluate the function at runtime. Atruntime, the return value of FROM_BASE64 cannot be used anymore toalter the SQL statement, because that stage is long past. At this pointin time, the return value is harmless.

Software I used

$ uname -aLinux cirrus7 5.5.3-arch1-1 #1 SMP PREEMPT Tue, 11 Feb 2020 15:35:41 +0000 x86_64 GNU/Linux$ mysql --versionmysql  Ver 15.1 Distrib 10.4.12-MariaDB, for Linux (x86_64) using readline 5.1$ zenity --version3.32.0

Skipping or Selectively Running Tests With Maven

Tue, 25 Feb 2020 20:59:17 +0100

You might not always want to execute all tests. Sometimes you only want to runa single unit test, or only integrations tests that start with Foo. Let’s lookat how to make this happen in Maven.

Skip Unit or Integration Tests

Skip the execution of unit tests:

$ mvn compile -DskipTests

This will still compile the unit tests, but not run them. To skip bothcompilation and execution, use -Dmaven.test.skip. Note that thissetting is also honored by the Failsafe plugin. Skip executingintegration tests:

$ mvn verify -DskipITs

Execute a Single Unit or Integration Test Class

Execute a single unit test class with Surefire:

$ mvn test -Dtest=MyUnitTest

In a multi-module project, we need to add -DfailIfNoTests=false,otherwise a module will fail where a class with that name is notavailable. Also note that we do not execute integration tests because weinstruct Maven to stop after the test phase. To run a single unit testclass and skip all integration tests, add -DskipITs. Execute a singleintegration test class with Failsafe:

$ mvn verify -Dit.test=MyIntegrationTest

As you can guess by now, this will run the single integration test bythe name of MyIntegrationTest, and will also run all unit tests. Toprevent this, you can add -DskipTests. Again, don’t forget to set-DfailIfNoTests if you run Maven on a multi-module project.

Execute a Single Unit or Integration Test Within a Test Class

Both the Surefire and Failsafe Maven plugins allow you to go crazy intheir syntax for filtering which individual test classes or methodsshould be run. Specifying the parameters -Dtest and -Dit.test willoverride all defined include all defined includes and excludes from yourpom.xml. Maven will create an include that looks like**/${test}.java. We have seen how we can run single test classes byspecifying the class name after -Dtest or -Dit.test. The next mostsimple format (since version 2.7.3) takes the form ofClassName#methodName, so if you have a test class namedOrderTest.java and a unit test method testEmptyOrderShouldThrow, youcan run this test method by itself, excluding all other test by typing:

$ mvn test -Dtest=OrderTest#testEmptyOrderShouldThrow

Complexer syntax with both glob and regex patterns is possible sinceversion 2.19. The documentation gives this example:

$ mvn test '-Dtest=???Test, !Unstable*, pkg/**/Ci*leTest.java,> *Test#test*One+testTwo?????, #fast*+slowTest'

Let’s break this down. As you can see, we specify multiple patterns in asingle parameter (to prevent shell expansion, they are put inside singlequotes):

???Test will run any test that has a name that starts with anythree characters and ends with Test (? is a glob charactermeaning any single character).
!Unstable* will not run any test that starts with Unstable (theinitial ! negates the pattern).
pkg/**/Ci*leTest.java will run any test that has pkg somewherein its package name, followed by any number of other child packagenames, while the class name should start with Ci followed by zeroor more wildcard characters and ending with leTest.java(pkg/CileTest.java matches, as doesfoo/bar/pkg/baz/CiqwertyleTest.java).
*Test#test*One+testTwo????? will run any test whose class nameends in Test and whose method name matches test + any amount ofcharacters + One or testTwo + any five characters. So this willrun two test methods in every test class called *Test.
#fast*+slowTest will run the methods fast* and slowTest whereever there are found.

We can even have more fun by throwing in regular expressions:

$ mvn test '-Dtest=Basic*, !%regex[.*.Unstable.*],> !%regex[.*.MyTest.class#one.*|two.*], %regex[#fast.*|slow.*]'

!%regex[.*.Unstable.*] will not run any tests that containUnstable somewhere in their path.
!%regex[.*.MyTest.class#one.*|two.*] will run methods named oneand two in a class called MyTest. We see that when using globs,we can use forward slashes to delimit packages. However, when usingregex, we use dots.
%regex[#fast.*|slow.*] will run the test methods fast and slowwherever they are found. Note that (at least in version 3.0.0-M4),the trailing .* wildcard pattern is not needed.

Grokking Time Zones in Oracle's DATE and TIMESTAMP

Sat, 22 Feb 2020 18:22:14 +0100

This post talks a little bit about DATE and TIMESTAMP in Oracle and some oftheir pitfalls. TL;DR: Representing dates and timestamps correctly is a very hardproblem (required viewing).

Intro

When I was tasked with putting some data in BigQuery, Google Cloud’sdata warehouse implementation, I had to look at the data we currentlystore in our local Oracle database. I found that we were using DATEfor fields that store “moments in time”. Normally, I would expect Unixtimestamps here, or some other data type that contains the time zone.

Naturally, I complained about this, but people were quick to point outthat it doesn’t matter, because you should relate the DATE to the timezone of the database. Oracle, it turns out, is tracking two time zones:the time zone of the database itself:

SELECT dbtimezone  FROM dual;-- Europe/Amsterdam

and the time zone of the client session:

SELECT sessiontimezone  FROM dual;-- Europe/Moscow

(See here how to update these timezones.)

I was warned that Oracle might change the query results based on eitherof these time zones, meaning when you insert a data type related todates or time, you might get back an altered representation based on thetime zone of the database or the session of the client.

This did not put my mind at ease at all. So, let’s take a look at whatsome of the common types relating to date and time are in Oracle, andwhen we should be careful with the query results.

`DATE`s

A DATE is a calendar date with a wallclock time without a time zone.If you put 2020-02-20 09:11:43 in the database, you will alwaysretrieve the same exact datetime, no matter whether you’re in Amsterdamor New York. (It is the equivalent of ajava.time.LocalDateTime,if that’s your thing.)

SELECT sessiontimezone  FROM dual;-- Europe/AmsterdamSELECT def_time  FROM users WHERE username = 'foo@example.com'-- 2020-02-20 09:04:50ALTER SESSION SET time_zone = 'UTC';SELECT def_time  FROM users WHERE username = 'foo@example.com'-- 2020-02-20 09:04:50

Plain `TIMESTAMP`s

A TIMESTAMP (without a time zone) is the same beast as a DATE,except that you can specify times up to a fraction of a second (a DATEneeds to be rounded to the nearest second).

`TIMESTAMP WITH LOCAL TIME ZONE`

Say, you store the value 2020-02-20 09:11:43 +01:00. A client with asessiontimezone of Europe/Amsterdam in the winter will retrieve thisexact value whether or not they retrieve it from an Amsterdam, Sydney orNew York database. Another client with a session time zone that is setto America/New_York, however, will get the value 2020-02-20 03:11:43 -05:00.

So: LOCAL means it adjusts the retrieved value to your session timezone.

ALTER SESSION SET time_zone = 'Europe/Moscow';  -- offset +03:00SELECT to_char("ttz", 'yyyy-MM-dd HH24:MI:SS')FROM (-- timestamp with local, New York time zone      SELECT CAST(              to_timestamp_tz('2020-02-20 09:11:43 -05:00',                              'yyyy-MM-dd HH24:MI:SS TZH:TZM')              AS TIMESTAMP WITH LOCAL TIME ZONE) AS "ttz"      FROM dual);-- 2020-02-20 17:11:43

`TIMESTAMP WITH TIME ZONE`

This data type stores the time zone together with the datetime anddisplays it independently from the session time zone of the client.

If we store a value 2020-02-20 09:11:43 +01:00 in Amsterdam and copythis to a database in New York, this is the value that is returned toclients in both Amsterdam and New York regardless of their session timezone.

Conclusion

DATEs and plain TIMESTAMPs don’t store a time zone and queries willalways return the exact same value that was inserted regardless ofdbtimezone or sessiontimezone. This makes them safe choices if yousomehow don’t care about time zones.

TIMESTAMP WITH LOCAL TIME ZONE and TIMESTAMP WITH TIME ZONE dostore time zones. A TIMESTAMP WITH LOCAL TIME ZONE will be representeddifferently depending on your session time zone. A TIMESTAMP WITH TIME ZONE will not be represented differently; it will just include anoffset.

You might now always want to use TIMESTAMP WITH TIME ZONE, butapparently, they have problems of theirown.

Falling back to DATEs, however, is problematic because of DaylightSaving Time (DST) in the fall. When you store the current datetime inthe fall, and you go through the dance of setting all clocks back from3:00 to 2:00, you have no way of knowing at what exact point in time therow was inserted. This is bad if you rely on this for anything crucial,such as forensics (“so the hacker copied the entire database at 2:30,but was that before or after we reset our clocks?”).

Using Oracle `DATE`s in Java

As for me, having to deal with DATEs without an explicit time zone,this means that the application inserting/retrieving these datetimeswill need to decide how to interpret them. In my Java application, Iwill map them tojava.time.LocalDateTime,add a time zone and convert them tojava.time.Instants:

Instant defTimeInstant = user.getDefTime()        .atZone(ZoneId.of("Europe/Moscow"))        .toInstant();

user.getDefTime() returns a LocalDateTime indicating when the userwas created. We can give a LocalDateTime a time zone by invokingatZone(ZoneId) on it, returing aZonedDateTime.Once we have converted the datetime to an actual point on the timeline(no longer being some kind of abstract datetime without a timezone), wecan convert it to an Instant by invoking toInstant() on theZonedDateTime.

Sources

Oracle documentation on datetime datatypes.

Oracle Sqlplus Cheat Sheet

Sat, 22 Feb 2020 16:30:58 +0100

In this post, I’m going to aggregate all those Oracle commands that I can neverremember but are very useful to have somewhere written down.

Intro

Last week, I suddenly had to work with an Oracle database again. Inormally use Intellij’s DataGripto connect to databases. I tried it this time, and I found I could notconnect to the schema I wanted: the schema just turned up empty. Ofcourse, everybody will recommend you use Oracle’s SQL Developer with anyOracle database you have to touch. So, after trying brew search sqldeveloper (yes, I’m on a Mac at work), coming up empty, readingthis caskrequest andfeeling the anticipation of endless frustration grow inside me, I wentto Oracle’s web site to see if I could download the program. I can,except that they want me to turn in a DNA sample first:

Of course, faking those kind of details is not impossible, but thehassle of going through something like that just for a lousy program soI can just run a couple of lousy queries puts me off. Luckily, I had aDocker container with Oracle XE lying around. It includes SQL*Plus, the“venerable” command-line tool provided by Oracle to query and administerOracle databases. Being a Arch Linux user with a predilection foranything with a command-line interface, it was not too bad, but gettingthe output formatted so that I could actually make sense of it requiredmore effort than most users are willing to put up with. So how do youfind your way around when you have SQL*Plus and no clue how it works?How do you find the schema’s you want, the tables you need to query? Howcan you check your privileges? This post aims to be a cheat sheet so youcan find your way around. Keep in mind that the results you’re gettingdepend on the privileges your user has. Getting an empty result doesnot mean that the database object does not exist. Rather, it meansyou’re a lesser god.

Getting Help With `sqlplus`

sqlplus does not have a man page, but provides help when you pass-h/-help:

$ sqlplus -h

Connecting to an Oracle Database Using SQL*Plus

The basic syntax to connect as user alice with password qwerty to adatabase FOO which is located on db.domain.tld and listens on port1521 (default port) is:

$ sqlplus alice/qwerty@db.domain.tld:1521/FOO

Show the Connected User

SHOW USER;

The SHOW command lets you look at the current state of your SQL*Plusenvironment: SHOW ALL shows all settings.

Show All Schema’s

SELECT username  FROM dba_users u;

Return only non-empty schema’s (excluding most users who never createdany object in the database):

SELECT username  FROM dba_users u WHERE EXISTS (-- filter users without database objects               SELECT 1               FROM dba_objects o               WHERE o.owner = u.username);

Excluding Oracle’s built-in schema’s:

SELECT username  FROM dba_users WHERE default_tablespace NOT IN ('SYSTEM', 'SYSAUX');

Source

Show All Tables/Views in a Particular Schema

SELECT table_name  FROM all_tables WHERE owner = 'MYSCHEMA';

Related: find all views:

SELECT view_name, text  FROM all_views WHERE owner = 'MYSCHEMA';

Describe a Table

DESCRIBE table;

Show DDL of a Table

SELECT dbms_metadata.get_ddl('VIEW', 'USERS', 'MYSCHEMA')  FROM dual;

where the first argument is the type of object (e.g. 'TABLE','VIEW', 'TRIGGER'), the second is the name of the object, and thethird the schema where the object is defined.

Show the Privileges of Your User

SELECT * FROM USER_SYS_PRIVS;SELECT * FROM USER_TAB_PRIVS;SELECT * FROM USER_ROLE_PRIVS;

Source

Get More Decent Output From a Command

If you want SQL*Plus to truncate the value of a column:

SET WRAP OFF;

otherwise it will allow the column to wrap to the next line (defaultON). Suppress all headings, page breaks, titles, the initial blankline and other formatting information:

SET PAGESIZE 0;

SourceNow, if only Oracle would support \G for vertical output…

Get and Alter Database Timezones

SELECT dbtimezone FROM dual;ALTER DATABASE SET dbtimezone = 'Europe/Amsterdam';

SELECT sessiontimezone FROM dual;ALTER SESSION SET sessiontimezone = 'Europe/Amsterdam';

Select `DATE` Rows in the Last Hour

Table t has a column c_date of type DATE:

SELECT *  FROM table t WHERE c_date > sysdate - 1/24;

This works because you can subtract a fraction from a DATE type wherethe fraction is interpreted as a fraction of a day (1 is an entireday, 0.5 is 12 hours, etc.).

Working Around Oracle Not Having a `LIMIT`

Yes, Oracle does not have a LIMIT keyword, so this idiom will quicklybecome ingrained in your muscle memory:

SELECT *FROM (-- actual query      SELECT firstName, lastName      FROM users)WHERE ROWNUM <= 10;

For each row returned by a query, the ROWNUM pseudocolumn returns anumber indicating the order in which Oracle selects the row from atable or set of joined rows. The first row selected has a ROWNUM of1, the second has 2, and so on. — Oracledocumentation

You could do without the subquery, plainly writing:

SELECT firstName, lastNameFROM usersWHERE ROWNUM <= 10;

but if you add a ORDER BY clause, Oracle will first select the firstten rows, and only then apply the ORDER BYclause, which might not be what you want. So that’s why it’s best toalways use the first idiom above.

Setting New Password for User

ALTER USER user_name IDENTIFIED BY new_password;

Show Output from Script

When writing a script, you may want to output some diagnostic data:

BEGIN    EXECUTE IMMEDIATE 'GRANT INSERT ON my.table TO role';EXCEPTION    WHEN OTHERS        THEN            IF SQLCODE = -942            THEN                DBMS_OUTPUT.PUT_LINE('Table does not exist, buddy');            ELSE                RAISE;            ENDIF;END;/

You think you’re good to go, but when you execute your script in an environmentwhere my.table does not exist, you don’t see the diagnostic message. Whatgives? SQL*Plus’s default behavior is to suppress output by default. You haveto SET SERVEROUTPUT ON first.

Mariadb/Mysql Password Exposure in Bash

Sun, 16 Feb 2020 19:07:46 +0100

How can we securely provide a password to mysql without exposing it to theworld by just putting it directly in the command?

How to Provide a Password to `mysql`?

Recently, I was writing a small Bash program with a GUI (usingzenity) that would promptthe user for their MariaDB credentials and then retrieve someinformation. Passing username and password to the mysql program lookslike:

$ mysql -u user -h host -ppassword mydatabase -e 'select 1;'

(Note that there cannot be a space between -p and the beginning of thepassword.)

Now, this obviously works fine, except for two problems:

We cannot reuse the connection to the database, so we have to make anew connection every time we want to execute a query.
The password of user user is given in the command itself. Thismeans that everyone on the same machine can list the processes andsee the command-line arguments passed to mysql.

In this post, I want to talk about the second problem.

$ pgrep -af mysql8046 mysql -u user -h 192.168.123.5 -px xxxxxxxxxxxxxxxxxxxxxxx mydatabase$ ls -l /proc/8046/cmdline-r--r--r-- 1 neftas neftas 0 16 feb 10:33 /proc/8046/cmdline$ cat -et /proc/8046/cmdlinemysql^@-u^@stefan^@-h^@192.168.123.5^@-px^@xxxxxxxxxxxxxxxxxxxxxxx^@mydatabase^@

As we can see, mysql replaces the provided password with xes, but atthe same time warns in man mysql that

Specifying a password on the command line should be consideredinsecure. You can use an option file to avoid giving the password onthe command line.

The option file man mysql is talking about is either one of theconfiguration files located at $HOME/.my.cnf or /etc/my.cnf, or acustom one specified on the command line. So, one option would be to putthe credentials in $HOME/.my.cnf before executing the program,stopping this issue cold in its tracks.

But let’s say that we want to dynamically connect to a database. Askingthe user to provide credentials and then write them to a file wouldprovide the same issues as passing it directly to the mysql command.The only difference would be that we now pass the sensitive informationto different command:

$ printf '[client]\nhost=%s\nusername=%s\npassword=%s\n' \    host username password > creds.tmp$ mysql --defaults-extra-file=creds.tmp mydatabase

The Environment Solution

Although not mentioned in my version of man mysql (10.4 from 29 March2019), the safest way, it turns out, is to set an environment variablenamed MYSQL_PWD. Why is this safer?

$ ls -l /proc/8046/environ-r-------- 1 neftas neftas 0 16 feb 11:00 /proc/8046/environ

At least on Linux, environment variables are only readable by the userthat started the process. This means other users (beware of root)cannot look at them.

To summarize:

declare -x MYSQL_PWDIFS='|' read -r username MYSQL_PWD < <(zenity --password --username)mysql -u "$username" -h host mydatabase

declare -x declares a variable that is going to be exported onassignment. This eliminates the need to export MYSQL_PWD explicitlysomewhere down the road: when MYSQL_PWD gets a value assigned, it willexport that value.

In the second line, I use process substitution (see Process Substitution in man bash) to provide a pipe-separated string toread that will split the string and assign the values to usernameand MYSQL_PWD.

Now that MYSQL_PWD is set, we no longer have to worry about providingthe password directly to the mysql command.

Versions Used In This Post

This post was written using the following versions:

$ uname -aLinux cirrus7 5.5.3-arch1-1 #1 SMP PREEMPT Tue, 11 Feb 2020 15:35:41 +0000 x86_64 GNU/Linux$ mysql --versionmysql  Ver 15.1 Distrib 10.4.12-MariaDB, for Linux (x86_64) using readline 5.1$ zenity --version3.32.0

Whats the Difference Between a Login and a Non-Login Shell?

Wed, 23 Jan 2019 20:37:44 +0100

I always kind of knew there was a difference between login and non-login shells,but when I couldn’t explain the difference properly to a coworker, I knew Ineeded to spend some time on figuring it out.

The difference is addressed nicely in the book Unix Power Tools (Oreilly):

When you first log in to a Unix system from a terminal, the systemnormally starts a login shell. A login shell is typically the top-levelshell in the “tree” of processes that starts with the init process.Many characteristics of processes are passed from parent to childprocess down this “tree” — especially environment variables, such asthe search path. The changes you make in a login shell will affect allthe other processes that the top-level shell starts — including anysubshells.
So, a login shell is where you do general setup that’s done only thefirst time you log in — initialize your terminal, set environmentvariables, and so on. […]

So you could think about a login shell as a shell that is started atstartup by the init process (or systemd nowadays). Or as a shellthat logs you into the system by your providing a username and apassword. A non-login shell, by contrast, is a shell that is invokedwithout logging anybody in.

There are two ways to check if your current shell is a login shell:First, you can check the output of echo $0: if it starts with a dash(like -bash), it’s a login shell. Be aware, however, that you canstart a login shell with bash --login, and echo $0 will output justbash without the leading dash, so this is not a surefire way of findout if you are running a login shell.

Secondly, the Unix StackOverflow offers this way of findingout:

$ shopt -q login_shell && echo login || echo nonlogin

(-q suppresses the output of the shopt command.)

Practically speaking, the difference between a login shell and anon-login shell is in the configuration files that Bash reads when itstarts up. In particular, according to man bash:

[…] it first reads and executes commands from the file/etc/profile, if that file exists. After reading that file, it looksfor ~/.bash_profile, ~/.bash_login, and ~/.profile, in thatorder, and reads and executes commands from the first one that existsand is readable.

You can observe this behavior by putting echo commands in/etc/profile, ~/.bash_profile, ~/.bash_login and ~/.profile.Upon invoking bash --login you should see:

echo from /etc/profileecho from ~/.bash_profile$

If the shell is a non-login shell, Bash reads and executes commands from~/.bashrc. Since we are starting a non-login shell from within a loginshell, it will inherit the environment. Sometimes, this will lead toconfusion when we inadvertently get a login shell, and find out that ourconfiguration from ~/.bashrc is not loaded. This is why many peopleput something like the following in their .bash_profile:

[[ -r ~/.bashrc ]] && source ~/.bashrc

This test whether .bashrc is readable and then sources it.

When you switch users using su you will take the environment of thecalling user with you. To prevent this, you should use su - which isshort for su --login. This acts like a clean login for a new user, sothe environment will not be cluttered with values from the calling user.Just as before, a login shell will read /etc/profile and the.bash_profile of the user you are switching to, but not its .bashrc.This post onStackOverflow shows whyyou might want to prefer to start with a clean environment (spoiler:your $PATH might be “poisoned”).

Conclusion

In this article we saw that the main difference between a login and anon-login shell are the configuration files that are read upon startup.We then looked at what the benefits are of a login shell over a non-loginshell.

Encrypt Device With Veracrypt From the Command Line

Sun, 06 Jan 2019 21:23:29 +0100

You have a drive that you want to encrypt and use in Linux and otherOSes. Then Veracrypt, the successor of Truecrypt, is a good choice. In thistutorial, I will show you how to quickly encrypt a drive and mount and unmountit from the command line.

The prerequisite for this tutorial is that you already have created a partitionon a drive. See my previous blogpost on how to accomplish that.Creating a volume on a partition with data on it will permanently destroythat data, so make sure you are encrypting the correct partition (fdisk -l isyour friend).

Encrypt a volume interactively from the command line using Veracrypt…

(The # sign at the beginning of the code examples indicates that thecommand should be executed as root. You can either use su - or sudoto accomplish this.)

# veracrypt -t --quick -c /dev/sdXX

-t is short for --text (meaning you don’t want the GUI) and shouldalways be used first after the command name. The --quick option isexplained in thedocs:

If unchecked, each sector of the new volume will be formatted. Thismeans that the new volume will be entirely filled with random data.Quick format is much faster but may be less secure because until thewhole volume has been filled with files, it may be possible to tellhow much data it contains (if the space was not filled with randomdata beforehand). If you are not sure whether to enable or disableQuick Format, we recommend that you leave this option unchecked. Notethat Quick Format can only be enabled when encryptingpartitions/devices.

So, using --quick is less secure, but not specifying it could take (alot) longer, especially on traditional hard drives (we’re talking hoursfor 500GB).

Finally, the -c or --create command allows us to specify on whichpartition we want to create a VeraCrypt volume. Make sure you change the/dev/sdXX from the example above to the appropriate output of fdisk -l (for example, /dev/sdc1).

This command will interactively guide us to create a new volume:

Volume type: 1) Normal 2) HiddenSelect [1]: 1Encryption Algorithm: 1) AES 2) Serpent 3) Twofish 4) Camellia 5) Kuznyechik 6) AES(Twofish) 7) AES(Twofish(Serpent)) 8) Camellia(Kuznyechik) 9) Camellia(Serpent) 10) Kuznyechik(AES) 11) Kuznyechik(Serpent(Camellia)) 12) Kuznyechik(Twofish) 13) Serpent(AES) 14) Serpent(Twofish(AES)) 15) Twofish(Serpent)Select [1]: 1Hash algorithm: 1) SHA-512 2) Whirlpool 3) SHA-256 4) StreebogSelect [1]: 1Filesystem: 1) None 2) FAT 3) Linux Ext2 4) Linux Ext3 5) Linux Ext4 6) NTFS 7) exFATSelect [2]: 6Enter password:WARNING: Short passwords are easy to crack using brute force techniques!We recommend choosing a password consisting of 20 or more characters. Are you sure you want to use a short password? (y=Yes/n=No) [No]: yRe-enter password:Enter PIM:Enter keyfile path [none]:Please type at least 320 randomly chosen characters and then press Enter:Characters remaining: 4Done: 100.000%  Speed: 61.8 GB/s  Left: 0 sThe VeraCrypt volume has been successfully created.

The volume is now created in the partition and is ready to be mounted.

… Or do it all in a one-liner

# veracrypt --text --quick                      \        --non-interactive                       \        --create /dev/sdXX                      \        --volume-type=normal                    \        --encryption=AES                        \        --hash=SHA-512                          \        --filesystem=NTFS                       \        --password='Un$@f3'

Use --stdin to read the password from the standard in, instead ofsupplying it directly to the command, which is considered unsecure.

Mounting the volume

Linux:

# mkdir /tmp/vera# veracrypt -t /dev/sdXX /tmp/vera

Windows:

>:: first find the VolumeName of the partition> mountvol.exe... snip ...Possible values for VolumeName along with current mount points are:    \\?\Volume{3676a1ae-0000-0000-0000-100000000000}\        *** NO MOUNT POINTS ***    \\?\Volume{1b98f0ba-8bc1-b740-b21f-f570bf2367dd}\        E:\    \\?\Volume{3676a1ae-0000-0000-0000-300300000000}\        C:\    \\?\Volume{3676a1ae-0000-0000-0000-c0a01f000000}\        *** NOT MOUNTABLE UNTIL A VOLUME MOUNT POINT IS CREATED ***    \\?\Volume{813379b4-3e59-11eb-bbcd-806e6f6e6963}\        D:\New volumes are not mounted automatically when added to the system.  To mount avolume, you must create a volume mount point.>>:: in my case, the VeraCrypt partition is mounted at E:>:: I'll make it available decrypted at Z:>>VeraCrypt.exe /v \\?\Volume{1b98f0ba-8bc1-b740-b21f-f570bf2367dd}\ /l z /q

At this point, a dialog shows where you can enter you password. I wouldn’trecommend it, but you can also specify /p <password> or /password <password> on the command line and skip the dialog.

The /q or /quit option makes sure the main VeraCrypt window is notdisplayed.

Of course, using the GUI makes all of this even simpler, as you don’t have tobother with finding the VolumeName yourself.

Unmounting the volume

Linux:

# veracrypt -d /tmp/vera

Windows:

>VeraCrypt.exe /d Z:

More info

Linux:

$ veracrypt -t -h

-h is short for --help and should be self-explanatory.

Windows:

>VeraCrypt.exe /help

Or read about the command-line options for Windows online.

Make Less Options Permanent or the Missing Lessrc

Sun, 06 Jan 2019 20:15:50 +0100

Let’s have a look at how we can make certain less preferences permanent,like -I, for example, which will make search case insensitive.

The missing `$HOME/.lessrc`

In GNU/Linux, preferences are often stored in rc files. For vim we have.vimrc, for Bash .bashrc, etc:

$ find "$HOME" -maxdepth 1 -name '*rc'./.vimrc./.idlerc./.xinitrc./.lynxrc./.old_netrc./.inputrc./.bashrc./.rtorrent.rc./.sqliterc./.xdvirc

Environment variable `LESS`

So, it would make sense to expect a .lessrc. But there is none.Instead, we define a environment variable LESS. My .bashrc:

export LESS="IFRSX"

Breakdown:

-I: ignore case when searching
-F: quit immediately when the entire file fits in one screen (ineffect, mimic cat’s behavior)
-R: enable colored output (for example, when piping to less fromdiff --color=always)
-S: truncate long lines instead of wrapping them to the next line
-X: don’t clear screen on exit

See man 1 less for all options.

Make a Backup With Rsync

Sun, 06 Jan 2019 17:21:58 +0100

We want to make a backup of data, for example to an external hard drive. Rsyncto the rescue.

The basic command

Assuming we are in someone’s home directory and we want to copy threesource directories Music, Documents and Movies to a destinationdirectory /mnt/external-hdd:

$ rsync -a Music Documents Movies /mnt/external-hdd

A word on slashes

Notice that we omit the trailing forward slash / on the sourcedirectories. This means the destination will look like:

/mnt/external-hdd|-- Music|   |-- a.mp3|-- Documents|   |-- b.txt|-- Movies|   |-- c.mp4

If we were to add trailing forward slashes, the upper-level sourcedirectories would not be copied and the result would look like:

/mnt/external-hdd|-- a.mp3|-- b.txt|-- c.mp4

The `rsync -a` command broken down

rsync -a is equal to rsync --archive and is a convenience command.According to the man page, it equals rsync -rlptgoD.

-r or --recurse: recursively copy data
-l or --links: copy symlinks as symlinks
-p or --perms: preserve permissions
-t or --times: preserve modifications times
-g or --group: preserve the group
-o or --owner: preserve owner
-D is the same as --devices --specials:
--devices: preserve device files
--specials: preserve special files

[…] a device file or special file is an interface to adevice driver that appears in a file system as if it were an ordinaryfile
— Wikipedia

Device files in Linux are usually found under the /dev directory.

See the overall progress of `rsync`

By default, rsync will show the progress of the individual files thatare being copied. If you want the overall progress, you have to add someflags:

$ rsync -a --info=progress2 --no-i-r src dst

--info=progress2 shows the total transfer progress. (To see allavailable options for --info, execute rsync --info=help). --no-i-ris short for --no-inc-recursive and disables incremental recursion,forcing rsync to do a complete scan of of all directories beforestarting the file transfer. This is needed to get an accurate progressreport, otherwise rsync doesn’t know how much work is left.

Human-readable output can be obtained by passing the -h or--human-readable option.

For a discussion of these options, see also this StackOverflowpost.

Partition and Format Drive With NTFS

Sat, 05 Jan 2019 20:11:43 +0100

Say we bought an external hard drive to back up some stuff from acrashed computer. We can use a Live USB to get at the data and put thedata on the external hard drive. Because the data needs to be accessibleby Windows, we are going to use format the drive with NTFS.

Create partition

Connect the external hard disk to your computer. Use sudo fdisk -l tofind the device name. Output should look something like this:

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectorsDisk model: CT2000MX500SSD1Units: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 4096 bytesI/O size (minimum/optimal): 4096 bytes / 4096 bytesDisklabel type: dosDisk identifier: 0x117d68c1Device     Boot Start        End    Sectors  Size Id Type/dev/sdb1        2048 3907029167 3907027120  1.8T 83 Linux

As can been seen above, the name of the device is /dev/sdb. We use toname to run fdisk:

$ sudo fdisk /dev/sdb

Notice how we use the name of the device, and not the name of thepartition (so /dev/sdb without any numbers attached at the end).

After entering the command above, an interactive menu will be facingyou. Type a letter and press Enter to confirm.Changes will only be applied when you type w, so if you make amistake, just stay calm and press q and you will exit fdisk withyour pending changes discarded.

Delete all your existing partitions by pressing d. Depending onthe amount of partitions, you might have to repeat this severaltimes. If you want to check the current partition table, press p.
After all old partitions are deleted, add a new partition bypressing n. If you just want to create a single partition on yourdrive, accept all the defaults by pressingEnter on each prompt. This will leave youwith a single partition that will take up all space on the drive.
Back in the main menu, type t to change the partition type. PressL to see all partitions types. Here we are going to choose 7(HPFS/NTFS/exFAT). “The partition type […] is a byte valueintended to specify the file system the partition contains and/or toflag special access methods used to access these partitions”(source). Linuxdoes not care about the partitiontype,but Windows does, so we have to change it.
Press w to write your changes to the disk and exit fdisk.

Format partition with NTFS

Now we create the actual NTFS file system on the drive:

$ sudo mkfs.ntfs -Q -L label /dev/sdX1

(If you don’t have mkfs.ntfs installed, use your distro’s packagemanager to install it (on Arch Linux it’s in a package calledntfs-3g)).

Breakdown:

-Q is the same as --quick, -f or --fast. This will perfom aquick format, meaning that it will skip both zeroing of the volumeand and bad sector checking. So obviously, leave this option out ifyou want the volume to be zeroed or you want error checking.Depending on the size of your partition, this might take quite awhile.
-L is the same as --label: it’s the identifier you’ll see inWindows Explorer when your drive is connected.
dev/sdX1: change the X to the actual letter of your drive wefound earlier in this tutorial. You always format a partition, not adrive, so make sure that you put the correct number of the partitionyou want formatted at the end.

Create Arch Linux Live USB

Sat, 05 Jan 2019 17:31:05 +0100

Creating a Arch Linux live USB is easy. In this post, I will walk you throughdownloading and verifying the image, finding your USB drive and copying theimage onto it, all from the convenience of the command line.

Download image and PGP signature

Download the latest Arch Linux image from https://www.archlinux.org/download/.The preferred option is to download the image using BitTorrent, in order not toburden the Arch servers unnecessarily.

Verify downloaded image

$ gpg --keyserver pgp.mit.edu                       \        --keyserver-options auto-key-retrieve       \        --verify archlinux-version-x86_64.iso.sig

If this disk is being created on an Arch Linux system, you could alsoinvoke:

$ pacman-key -v archlinux-version-x86_64.iso.sig

The -v switch is short for --verify.

Insert USB drive and check the device name

$ sudo fdisk -l

The -l is short for --list, and will display the device names andpartition tables.

Output will look like:

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectorsDisk model: CT2000MX500SSD1Units: sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 4096 bytesI/O size (minimum/optimal): 4096 bytes / 4096 bytesDisklabel type: dosDisk identifier: 0x117d68c1Device     Boot Start        End    Sectors  Size Id Type/dev/sdb1        2048 3907029167 3907027120  1.8T 83 Linux

Check the output and find the device name of the USB (for instance/dev/sdc). Make sure this device is not mounted, otherwise thenext command will fail. Also make sure you note the device name, andnot a partition (indicated by a numeral at the end: /dev/sdc1, forexample).

Copy Arch Linux image to USB drive

$ sudo dd if=archlinux-2019-01-01-x86_64.iso    \        of=/dev/sdX                             \        bs=64K                                  \        oflag=sync                              \        status=progess

Breakdown:

if indicates the input file (the .iso of the live Linux distro).
of, likewise, points to the output file, which is a device in thiscase. Note that /dev/sdX needs to be replaced with the device namewe found in the previous step.
bs=64K indicates the block size, which means that dd will readand write up to 64K bytes at a time. The default is 512 bytes. Itreally depends what the optimal block size is, but severalsourcesindicate that 64K is a good bet on somewhat modern to modernhardware.
oflag stands for “output flag”. The sync flag will make surethat all data is written to the USB stick when the dd commandexits, so it will be safe to remove the USB stick.
status=progress indicates the level of information that is printedduring file transfer. progress shows periodic transfer statistics.

Notice that the device does not need to be partitioned or empty beforethis operation. When dd writes to a device rather than a partition,all data on the drive – including partitions – will be erased anyway.

Spring Basics Conditional Bean Wiring

Fri, 19 Oct 2018 19:18:16 +0200

In Spring, it’s possible to wire a bean only when some condition is met.

A Use Case for Conditional Bean Wiring

Suppose we want to enable caching for our web application. We areinterested to find out whether this helps our application or whether theoverhead will so big that it actually slows our application down. Wedecide we want to put the new functionality behind a feature toggle. Weadd a new property to our application calledapp.caching.enabled=false. When we are ready to enable the caching, wechange the property’s value to true and we are in business. When theresults disappoint, we can easily revert the property’s value tofalse.

`@Conditional` Introduction

The @Conditional annotation can be used on any type or method thatdeclares a bean:

@Conditional(SomeCondition.class)@Beanpublic SomeBean someBean() {    return new SomeBean();}

@Conditionaltakes a mandatory class that implements the functional interfaceCondition.Condition defines a single method matches that returns a boolean.The method decides whether the bean should be loaded into the Springcontext or whether it should be ignored:

public SomeCondition implements Condition {    public boolean matches(ConditionContext context,                            AnnotatedTypeMetadata metadata) {        /* make decision based on context and metadata */    }}

A `@Conditional` `@Configuration`

When @Conditional is applied to a class which is also annotated with@Configuration, then all of the @Bean methods, @Import,@ComponentScan and other annotations will be subject to the condition.This means that when the condition evaluates to false, all of theconfiguration defined in that class will be ignored.

Since creating a cache carries costs with it (even an unused cachereserves space on the heap for its initial size), this is exactly whatwe want in our situation.

@Configuration@Conditional(CacheCondition.class)public class CacheConfig {    /* ... define cache manager and caches here ... */    public static CacheCondition implements ConfigurationCondition {        public ConfigurationPhase getConfigurationPhase() {            return ConfigurationPhase.PARSE_CONFIGURATION;        }        public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {            return context.getEnvironment().getRequiredProperty("app.caching.enabled", Boolean.class);        }    }}

Since we are dealing with a conditional configuration instead of aregular bean, the condition class implementsConfigurationConditioninstead of Condition. The ConfigurationCondition makes us implementanother method that specifies at which point the condition should beevaluated. Normally, thisConfigurationPhaseis set to REGISTER_BEAN, which means it checks the condition whenadding a regular bean. In this case, we want it set toPARSE_CONFIGURATION, which makes Spring evaluate the condition at themoment the configuration class is parsed. If the condition does notmatch at that moment, the @Configuration class will not be added tothe context.

Dependencies

The only dependency is on spring-context:

<dependency>    <groupId>org.springframework</groupId>    <artifactId>spring-context</artifactId>    <version>5.1.1.RELEASE</version></dependency>

Sample Project

A sample project can be found onGitLab.

Spring Basics Dynamically Inject Values With Springs Value

Sun, 09 Sep 2018 18:57:22 +0200

If we do not want to hard-code values into our source code, we can useproperties files. With the @Value annotation, Spring gives us an easymeans to get properties from properties files and inject them into ourcode.

Dependencies

This post was written with Spring 5.0.5.RELEASE and Java 1.8.0_181.

<dependency>    <groupId>org.springframework</groupId>    <artifactId>spring-context</artifactId>    <version>5.0.5.RELEASE</version></dependency>

Inject Scalars

Let’s say we have the following key-value pairs in a fileapp.properties:

app.string.property=helloapp.integer.property=987app.floating.point.property=3.14159app.boolean.property=false

To get access to these properties, we declare an applicationconfiguration in Java:

@Configuration@ComponentScan@PropertySource("app.properties")public class AppConfig {}

The @PropertySource adds a property source to Spring’s Environment.Here, the properties file is placed in the root of the classpath (andsince I am using Maven default paths, that would besrc/main/resources.

package com.relentlesscoding.wirebeans;import org.junit.Assert;import org.junit.Test;import org.junit.runner.RunWith;import org.springframework.beans.factory.annotation.Value;import org.springframework.test.context.ContextConfiguration;import org.springframework.test.context.junit4.SpringRunner;import static org.hamcrest.CoreMatchers.is;import static org.junit.Assert.assertThat;@RunWith(SpringRunner.class)@ContextConfiguration(classes = AppConfig.class)public class HabitTest {    @Value("${app.string.property}")    private String stringProperty;    @Value("${app.integer.property}")    private int integerProperty;    @Value("${app.floating.point.property}")    private float floatingPointProperty;    @Value("${app.boolean.property}")    private boolean booleanProperty;    @Test    public void stringProperty() {        assertThat(stringProperty, is("hello"));    }    @Test    public void integerProperty() {        assertThat(integerProperty, is(987));    }    @Test    public void floatingPointProperty() {        assertThat(floatingPointProperty, is(3.14159F));    }    @Test    public void booleanProperty(){        assertThat(booleanProperty, is(false));    }}

Injecting Complex Values

Let’s say we also have collections of values in our properties file:

app.collection.strings.property=one, two, threeapp.collection.floats.property=1.2, 3.4, 5.6app.collection.dates.property=2018-09-30T10:00:00, 2018-09-29T11:00:00, 2018-09-28T12:00:00

By default, Spring won’t be able to interpret Lists of values. Thebest we can do, by default, is getting an array of Strings:

@Value("${app.collection.strings.property}")private String[] stringArrayProperty;@Testpublic void stringsArrayProperty() {    assertArrayEquals(new String[]{"one", "two", "three"}, stringArrayProperty);}

Declare a `ConversionService` to Inject Collections And Other Complex Types

To convert properties to other types than strings, we need to enableSpring’sConversionService.

@Beanpublic static ConversionService conversionService() {    return new DefaultFormattingConversionService();}

We instantiate DefaultFormattingConversionService, which is“configured by default with converters and formatters appropriate formost applications”. That means that it can convert comma-separatedstrings to common, generic collection types, and can convert strings todates, currencies and TimeZones, for example.

Notice that the method is declared static. ConversionService is oftype BeanFactoryPostProcessor and must be instantiated very early inthe Spring container life cycle, so that its services are available whenprocessing of annotations such as @Autowired and @Value are done. Bydeclaring the method static, the ConversionService can be invokedwithout instantiating the enclosing @Configuration class. Therefore,we can use @Value (and other annotations) that make use of thisconverter in the same class, without running into the trouble thatConversionService is not yet instantiated.

To learn more about bean instantiation, read the part underBeanFactoryPostProcessor-returning @Beanmethods.Also, this answer onStackOverflow is very clarifying.

@Testpublic void floatArrayProperty() {    assertArrayEquals(new float[]{1.2F, 3.4F, 5.6F}, floatArrayProperty, 0.01F);}

To convert dates from a properties string to actual date objects we needan extra step: @DateTimeFormat. This annotation indicates the formatof the date to Spring. In this case, we format our property strings toconform to ISO-8601 or DateTimeFormat.ISO.DATE_TIME:

@DateTimeFormat(iso = DateTimeFormat.ISO.DATE_TIME)@Value("${app.collection.dates.property}")private List<LocalDate> listOfDatesProperty;@Testpublic void listOfDatesProperty() {    assertThat(listOfDatesProperty.size(), is(3));    assertEquals(LocalDate.parse("2018-09-30T10:00:00", DateTimeFormatter.ISO_DATE_TIME), listOfDatesProperty.get(0));}

Thanks to our ConversionService, we can now convert not only toarrays, but also to collections.

Inject Maps With Spring’s @Value

If we throw in a Spring Expression Language (SpEL) expression, we caneven have dictionaries in our properties and convert them to Maps:

app.collection.map.string.to.integer={one:"1", two:"2", three:"3"}

@Value("#{${app.collection.map.string.to.integer}}")private Map<String, Integer> mapStringToInteger;@Testpublic void mapProperty() {    assertThat(mapStringToInteger.size(), is(3));    assertEquals(new Integer(1), mapStringToInteger.get("one"));    assertEquals(new Integer(2), mapStringToInteger.get("two"));    assertEquals(new Integer(3), mapStringToInteger.get("three"));}

Notice the #{...} that delimits the SpEL expression. It evaluates thestring that comes from the properties files and parses it to a Map. Tounderstand how this works, let’s have a look at what a literal map in aSpEL expression would look like:

@Value("#{{1: 'Catch-22', 2: '1984', 3: 'Pride and Prejudice'}}")private Map<Integer, String> books;

A literal map in a SpEL expression is delimited by braces {key: 'value', ...}. This is exactly what we had in our properties file.

Spring Basics XML Setter Injection With Custom Method Names

Sun, 09 Sep 2018 11:46:25 +0200

Let’s have a look at how to inject beans using setters with a custom methodname.

In a previous blogpost we ended upwith the following XML-based configuration:

<?xml version="1.0" encoding="UTF-8" ?><beans xmlns="http://www.springframework.org/schema/beans"       xmlns:util="http://www.springframework.org/schema/util"       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"       xsi:schemaLocation="http://www.springframework.org/schema/beans       http://www.springframework.org/schema/beans/spring-beans.xsd       http://www.springframework.org/schema/util       http://www.springframework.org/schema/util/spring-util.xsd">    <bean id="startTime" class="java.time.LocalDate" factory-method="now"/>    <bean id="firstStreak" class="com.relentlesscoding.wirebeans.PositiveStreak">        <constructor-arg ref="startTime"/>    </bean>    <bean id="secondStreak" class="com.relentlesscoding.wirebeans.PositiveStreak">        <constructor-arg ref="startTime"/>    </bean>    <util:list id="myRunningStreaks">        <ref bean="firstStreak"/>        <ref bean="secondStreak"/>    </util:list>    <bean id="running" class="com.relentlesscoding.wirebeans.Running">        <property name="streaks" ref="myRunningStreaks"/>    </bean></beans>

The complete code can be found onGitLab.

To recap, we instantiate a bean of type LocalDate by invoking itsnow() static method. We inject this bean into two PositiveStreakbeans using constructor injection. We reference the LocalDate bean byits id, startTime. Next, we create a bean that is a list ofPositiveStreaks using the util namespace. We then inject this listinto a Habit class called Running using setter injection.

For setter injection to work from XML, the setter method needs to followJava beanconventions.This means that there should be setter method calledset<PropertyName>, where the important part is that the method startwith set. The name of the property itself can be something completelydifferent from the field it is setting, Spring does not care. Forexample, the following would work without a hitch:

class Running implements Habit {    private List<Streak> myStreakList;    public void setStreaks(List<Streak> aListOfStreaks) {        this.myStreakList = aListOfStreaks;    }}

<bean class="com.relentlesscoding.wirebeans.Running">    <property name="streaks" ref="myRunningStreaks" /></bean>

The only important thing is that the name attribute of the propertyelement corresponds to the Property part of setProperty settermethod in Java. To reiterate: The name of the field that is being setby the mutator method can be something completely different from thename attribute of the property element.

But sometimes we want to be creative with our method names if that makesthe code more readable, or maybe we are dealing with code that we cannotchange. If we end up with a mutator method that does not start withset, you are now in a position to appreciate that Spring will not findthis method and throws a BeanCreationException.

Handling Mutators With a Non-Standard Names

On StackOverflow a solution wasproposedthat uses aMethodInvokingFactoryBean.But since we are invoking a method that does not return a result, thedocumentation recommends we useMethodInvokingBeaninstead.

Suppose we have the follow setter in place:

class Running implements Habit {    private List<Streak> myStreakList;    public void replaceStreaks(List<Streak> aListOfStreaks) {        this.myStreakList = aListOfStreaks;    }}

We could leverage a MethodInvokingBean like so:

<bean id="running" class="com.relentlesscoding.wirebeans.Running"><bean id="caller" class="org.springframework.beans.factory.config.MethodInvokingBean">    <property name="targetObject" ref="running"/>    <property name="targetMethod" value="replaceStreaks"/>    <property name="arguments" ref="myRunningStreaks"/></bean>

(Notice that we actually use setter injection on theMethodInvokingBean by specifying properties on it.) The targetObjectattribute specifies on which instance a targetMethod should beinvoked. We can specify our custom mutator name, and pass a reference toour arguments.

This is verbose, so think carefully before you violate the Java beannaming convention. Consider using afacade if the code isnot under your control. Alternatively, Java-based Spring configurationwith the @Autowired annotation will work on a method with any name.

Spring Basics Wiring Beans With Xml Configuration

Wed, 05 Sep 2018 19:52:43 +0200

If you have to work with legacy Spring applications, chances are youwill have to know how XML-based configuration works. Although Javaconfiguration is preferred for new applications, sometimes you justdon’t have a choice, so you’d better be comfortable with it.

The Code

You can find the code from this blog post onGitLab.

Dependencies

The only dependency you need to get a Spring container running isspring-context. Add in Maven:

<dependency>    <groupId>org.springframework</groupId>    <artifactId>spring-context</artifactId>    <version>5.0.5.RELEASE</version></dependency>

Declaring Beans in XML

The following class will become a bean:

package com.relentlesscoding.wirebeans;import java.util.List;public class Running implements Habit {    private final String name = "Running";    private final String description = "Run 10 km every day";    private List<Streak> streaks;    public String getName() {        return name;    }    public String getDescription() {        return description;    }    public List<Streak> getStreaks() {        return streaks;    }    public void setStreaks(List<Streak> streaks) {        this.streaks = streaks;    }    public void addStreak(Streak streak) {        streaks.add(streak);    }}

Create an XML file, give it any name (I would call itapplicationContext.xml) and place it in your src/main/resources.

<?xml version="1.0" encoding="UTF-8" ?><beans xmlns="http://www.springframework.org/schema/beans"       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"       xsi:schemaLocation="http://www.springframework.org/schema/beans            http://www.springframework.org/schema/beans/spring-beans.xsd ">    <bean id="running" class="com.relentlesscoding.wirebeans.Running"/></beans>

This would wire a bean with the id running that is of type Running(which is a Habit). You can see why this is much more of a hassle toset up than using a simple @Component annotation on a class or a@Bean annotation in a @Configuration class. Those XML namespaces arenasty, but luckily most IDEs will help you with them.

Injecting Beans

To inject dependencies, we have two choices:

Constructor injection
Setter injection

Unlike with Java configuration, we cannot insert into fields when usingXML-based configuration.

Constructor Injection

Suppose we want to insert one bean into another bean, for instance aHabitRepository that persists habits to the database into aHabitService. By using the <constructor-arg /> element and the refproperty, we can accomplish this:

<bean id="habitRepository" class="com.relentlesscoding.wirebeans.HabitRepository" /><bean id="habitService" class="com.relentlesscoding.wirebeans.HabitService">    <constructor-arg ref="habitRepository" /></bean>

If we wanted to pass more arguments to the HabitService constructor,we must keep on eye on the order: it must be the same as the order inwhich they are declared in the class.

To ease working with constructor arguments and as a way to curtail theverbosity of the XML configuration, Spring offers the c XML namespaceto help wire beans without the need to create a sub-element<constructor-arg />:

<?xml version="1.0" encoding="UTF-8" ?><beans xmlns="http://www.springframework.org/schema/beans"       xmlns:c="http://www.springframework.org/schema/c"       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"       xsi:schemaLocation="http://www.springframework.org/schema/beans       http://www.springframework.org/schema/beans/spring-beans.xsd">    <bean id="currentTime" class="java.time.LocalDate" factory-method="now" />    <bean id="streak"        class="com.relentlesscoding.wirebeans.PositiveStreak"        c:startTime-ref="currentTime" />

We define a bean of type java.time.LocalDate and we use the staticfactory method now() to get an instance of it. We then use the cnamespace to pass it to the constructor of our PositiveStreak bean.c:startTime-ref="currentTime" should be read as: pass the reference tothe bean with id currentTime to the constructor argument that has thename startTime.

So we can reference constructor arguments by name. We can also referencethem by position. c:_0-ref="currentTime" would do the exact samething. XML does not allow a digit as the first character of anattribute, so we have to use an underscore. If there is only a singleargument to the constructor, we can even use the shorthandc:_-ref="currentTime". I would not want to promote this as readable,but it’s good to know it exists and might be used in the wild.

Setter Injection

To use setter injection with XML-based configuration, you use the<property> element. If you look back at the Running class above, yousee it has a method setStreak that takes a List<Streak>:

<bean id="startTime" class="java.time.LocalDate" factory-method="now" /><bean id="streak"    class="com.relentlesscoding.wirebeans.PositiveStreak"    c:_0-ref="startTime" /><bean id="running" class="com.relentlesscoding.wirebeans.Running">    <property name="streaks">        <list>            <ref bean="streak" />        </list>    </property></bean>

The <property> element has an attribute name that refers to thefield name of the bean being set. In the current case, class Runninghas a field named streaks. As a child element of <property> wedefine a list of Streak references.

For a list of literal String values, this would have looked like:

<property name="listOfStrings">    <list>        <value>string value 1</value>        <value>string value 1</value>        <value><null/></value>    </list></property>

The list is wired with literal Strings and even a literal null.

Spring also provides the p namespace to make this more convenient(less verbose):

<beans xmlns="http://www.springframework.org/schema/beans"       xmlns:c="http://www.springframework.org/schema/c"       xmlns:p="http://www.springframework.org/schema/p"       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"       xsi:schemaLocation="http://www.springframework.org/schema/beans       http://www.springframework.org/schema/beans/spring-beans.xsd">    <bean id="startTime" class="java.time.LocalDate" factory-method="now"/>    <bean id="streak"        class="com.relentlesscoding.wirebeans.PositiveStreak"        c:_0-ref="startTime" />    <bean id="running"        class="com.relentlesscoding.wirebeans.Running"        p:streaks-ref="streak" /></beans>

The usage of the p namespace is a lot like that of the c namespace.In this case, p:streaks-ref="streak" tells spring to wire a propertynamed streaks with the bean that is referenced by the id streak.Now, the property streaks takes a List. If we pass only a singleelement to that list, the current syntax works and Spring will happilyinsert the single reference to streak into a List for us and passthat to the setter method. If we want to pass more than one element in alist, however, we have to create the list separately first, and thenpass the id of that reference to the p property:

<beans xmlns="http://www.springframework.org/schema/beans"       xmlns:c="http://www.springframework.org/schema/c"       xmlns:p="http://www.springframework.org/schema/p"       xmlns:util="http://www.springframework.org/schema/util"       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"       xsi:schemaLocation="http://www.springframework.org/schema/beans       http://www.springframework.org/schema/beans/spring-beans.xsd       http://www.springframework.org/schema/util       http://www.springframework.org/schema/util/spring-util.xsd">    <bean id="startTime" class="java.time.LocalDate" factory-method="now"/>    <bean id="firstStreak" class="com.relentlesscoding.wirebeans.PositiveStreak" c:_0-ref="startTime"/>    <bean id="secondStreak" class="com.relentlesscoding.wirebeans.PositiveStreak" c:_0-ref="startTime"/>    <util:list id="myRunningStreak">        <ref bean="firstStreak"/>        <ref bean="secondStreak"/>    </util:list>    <bean id="running" class="com.relentlesscoding.wirebeans.Running" p:streaks-ref="myRunningStreak"/></beans>

We need to add the util namespace and the location of the utilschema definition to get this to work. You see that the XML becomesquite verbose the more you try to do with it. The util namespaceallows us to create collections of literal values or beans. Thesecollections can then be referenced by their id.

More information:

Taking the App for a Test Ride

package com.relentlesscoding.wirebeans;import org.junit.Assert;import org.junit.Test;import org.junit.runner.RunWith;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.test.context.ContextConfiguration;import org.springframework.test.context.junit4.SpringRunner;@RunWith(SpringRunner.class)@ContextConfiguration(locations = "classpath:applicationContext.xml")public class HabitTest {    @Autowired    Habit runningHabit;    @Test    public void runningHabitIsNotNull() {        Assert.assertNotNull(runningHabit);    }    @Test    public void runningHabitHasSingleStreak() {        Assert.assertEquals(2, runningHabit.getStreaks().size());    }}

To run integration tests where the Spring context is available, you needthe following dependency:

<dependency>    <groupId>org.springframework</groupId>    <artifactId>spring-test</artifactId>    <version>5.0.5.RELEASE</version>    <scope>test</scope></dependency>

spring-test contains the SpringRunner JUnit runner, and the@ContextConfiguration that will tell Spring where to look for theapplication context that contains the beans that need to be wired. Inthis case, we tell it to look at the applicationContext.xml that weput in src/main/resources, so we can reference it by looking at theroot of our classpath with the attribute locations = "classpath:applicationContext.xml".

Quick Vim Tip: Change Text Between White Space

Sun, 02 Sep 2018 17:00:22 +0200

private List<String> strings;

When the cursor is on the S of String, use ciW to deleteList<String> and start insert. Obviously, you can use any othercommand with that movement, for example yank the text (yiW) or deleteit (diW).

From :help WORD:

A WORD consists of a sequence of non-blank characters, separated withwhite space. An empty line is also considered to be a WORD.

Spring Basics Wiring and Injecting Beans With Java Configuration

Sun, 02 Sep 2018 16:48:04 +0200

For new projects, Java configuration is preferred over XML-basedconfiguration. In this post, we’re going to look at how to configure Spring withconfiguration in Java, instead of the traditional XML.

For XML-based configuration, see a future blog post.

The code

You can find the code from this blog post onGitLab.

Dependencies

The only dependency you need to get a Spring container running isspring-context. Add in Maven:

<dependency>    <groupId>org.springframework</groupId>    <artifactId>spring-context</artifactId>    <version>5.0.5.RELEASE</version></dependency>

Detecting beans by scanning for components

Create a class (which can have any name, here I chose AppConfig) andannotate it with @Configuration and @ComponentScan:

package com.relentlesscoding.wirebeans;import org.springframework.context.annotation.ComponentScan;import org.springframework.context.annotation.Configuration;@Configuration@ComponentScanpublic class AppConfig {}

By default, the @ComponentScan will recursively scan the package inwhich the AppConfig class is declared. To change this, you can add apackage to the value element(@ComponentScan("com.relentlesscoding.wirebeans.beans")) to scan onlythat package. If it troubles you that this is not type safe and hindersrefactoring (it is a simple string after all), you can also pass Classobjects to the basePackageClasses element. It will then scan thepackage that class is part of. Some people even recommend to createmarker interfaces in each package for this purpose, but I think thisclutters up the source code too much.

Three ways to declare beans

Now Spring is able to detect our beans. We can declare beans in threeways:

By annotating a class with @Component.
By annotating a method with a non-void return type with @Bean ina class annotated with @Configuration.
By annotating any method with a non-void return type with @Bean.

Automatic configuration with `@Component`

The simplest way to declare a bean is by annotating a class with@Component:

@Componentpublic class Running implements Habit {    private final String name;    private final String description;    private final List<Streak> streaks;    // accessors omitted for brevity}

If we do not specify the value element of @Component, the id of thebean will be the lowercase name of the class, in this case running. Wecan use this id elsewhere to specify this particular bean, shouldambiguities arise.

Explicit configuration in class annotated with @Configuration

If you have control over the beans you are creating, i.e. you arewriting the source code, you would always go with automaticconfiguration by annotating your bean classes with @Component. If youare creating a bean for a class from a library, you can define yourbeans in your AppConfig class:

@Configuration@ComponentScanpublic class AppConfig {    @Bean    public List<Streak> streaks() {        List<Streak> streaks = new ArrayList<>();        streaks.add(new PositiveStreak(LocalDate.now()));        return streaks;    }}

Here, we defined a bean of type List<Streak> that we can now injectinto any other bean by using the @Autowired annotation (see below).

Lite Beans

Actually, we can declare any method with a non-void return type to bea @Bean. If we declare a bean outside of a configuration class, itwill become a “lite bean”. Spring will still manage its life cycle andscope and we can still autowire the bean into other beans, but wheninvoking the method directly, it will just be a plain-old Java methodinvocation without Spring magic. (Normally, Spring would create a proxyaround the bean and all invocations would go through the Springcontainer. This would mean that by default only a single instance of thebean would exist, for example. In “lite” mode, however, the annotatedmethod is just a factory method, and will happily instantiate a newobject every time it is called.)

Using the declared beans

To use Spring’s dependency injection, you have a couple of options, allof which involve annotating a method or field with @Autowired. Bydefault, a matching bean of the specified type needs to exist in theSpring context or else Spring will throw an exception. To make theinjection optional, set the required element of @Autowired tofalse.

Constructor injection

For mandatory dependencies, you should use constructor injection.“Mandatory” means the bean would not make sense without the bean onwhich it depends. For example, a HabitService persists Habits to thedatabase. So a DAO or repository would be a mandatory dependency.

@Componentpublic class HabitService {    private final HabitRepository habitRepository;    @Autowired    public Running(HabitRepository habitRepository) {        this.habitRepository = habitRepository;    }    ...}

Field injection

Field injection should not be preferred, because it makes testing(e.g. with mocks) harder to pull off. In the following example weinject a dependency into a private field. When we would try to mock thedependency, we would have to deal with the access restriction.

@Componentpublic class HabitService {    @Autowired private HabitRepository habitRepository;    ...}

Setter injection

A third way to inject dependencies is through setter injection. Putting@Autowired on any method with one or more parameters will make Springlook for appropriate bean candidates in the Spring context.

@Componentpublic class Running implements Habit {    private final String name = "Running";    private final String description = "Run 10 km every day";    private List<Streak> streaks;    @Autowired    public setStreaks(List<Streak> streaks) {        this.streaks = streaks;    }    ...}

Taking the application for a test run

package com.relentlesscoding.wirebeans;import org.junit.Assert;import org.junit.Test;import org.junit.runner.RunWith;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.test.context.ContextConfiguration;import org.springframework.test.context.junit4.SpringRunner;@RunWith(SpringRunner.class)@ContextConfiguration(classes = AppConfig.class)public class HabitTest {    @Autowired    Habit runningHabit;    @Test    public void runningHabitIsNotNull() {        Assert.assertNotNull(runningHabit);    }    @Test    public void runningHabitHasSingleStreak() {        Assert.assertEquals(1, runningHabit.getStreaks().size());    }}

We can specify the application context by using the@ContextConfiguration annotation and filling in the classes element.The JUnit 4 annotation @RunWith specifies the SpringRunner.class(which is a convenience extension of the longerSpringJUnit4ClassRunner).

@ContextConfiguration is part of the spring-test library:

<dependency>    <groupId>org.springframework</groupId>    <artifactId>spring-test</artifactId>    <version>5.0.5.RELEASE</version>    <scope>test</scope></dependency>

Use Vidir to Quickly Edit Filenames in Your Editor

Mon, 20 Aug 2018 20:10:00 +0200

If you have installed moreutils (see below), you can type vidir toopen up the current working directory in your $EDITOR. You can use allthe power of your editor to edit and/or delete filenames anddirectories. Editing a line will rename the file or directory, deletinga line will remove the file or directory.

The following will list all your JPEG pictures in the current directoryin your editor:

$ vidir *.jpeg

vidir is not recursive by default: if you want to recursively editfilenames, you can do:

$ find -type f -name '*.jpeg' | vidir -  # take note of the trailing dash -

Deleting non-empty directories

When trying to delete a non-empty directory, vidir will complain:

/usr/bin/vidir: failed to remove ./non-empty-directory: Directory not empty

We can use find again:

$ ls -1 non-empty-dirfile1.txtfile2.txt$ find | vidir -1   ./non-empty-dir2   ./non-empty-dir/file1.txt3   ./non-empty-dir/file2.txt

When we delete all the files from the directory and the directoryitself, the directory will be deleted.

To see what vidir is actually doing, you can pass it the -v or--verbose flag:

$ find | vidir -v -removed './non-empty-dir/file2.txt'removed './non-empty-dir/file1.txt'removed './non-empty-dir'

How to install

In Arch Linux, you can install the moreutils package with sudo pacman -S moreutils. On Debian distros, you can run sudo apt install moreutils.

Use Pandoc With Pygments to Highlight Source Code

Sun, 12 Aug 2018 19:32:38 +0200

I am someone who has JavaScript disabled by default in his browser (Iuse uMatrix in Firefox for that).Only when I trust a site and I need to use functionality that trulydepends on JavaScript will I turn it on. This hopefully protects me frommost of the known and unknown bad stuff out there on the internet. Italso makes me appreciate people who go through the trouble of makingtheir webpages work without JavaScript.

Until recently, I used a JavaScript plugin on this blog to format sourcecode. This bothered me, since using JavaScript just to display somesource code seems like overkill and makes people have to turn onJavaScript in their browsers just to see the source code formattednicely. I wanted to do better than that.

The way I normally write my blog posts is, I start with a Markdownarticle and then use pandoc to convert it toHTML which I then copy and paste into Wordpress (if there is a betterway to do this, please contact me). I noticed pandoc provides a switch--filter where you can specify a executable that can transform thepandoc output. The only problem is, you have to write a filter.Luckily, I found a GitHub gistthat has already figured out how to write one. Here is some Haskell foryou:

import Text.Pandoc.Definitionimport Text.Pandoc.JSON (toJSONFilter)import Text.Pandoc.Sharedimport Data.Char(toLower)import System.Process (readProcess)import System.IO.Unsafemain = toJSONFilter highlighthighlight :: Block -> Blockhighlight (CodeBlock (_, options , _ ) code) = RawBlock (Format "html") (pygments code options)highlight x = xpygments:: String -> [String] -> Stringpygments code options         | (length options) == 1 = unsafePerformIO $ readProcess "pygmentize" ["-l", (map toLower (head options)),  "-f", "html"] code         | (length options) == 2 = unsafePerformIO $ readProcess "pygmentize" ["-l", (map toLower (head options)), "-O linenos=inline",  "-f", "html"] code         | otherwise = "<div class =\"highlight\"><pre>" ++ code ++ "</pre></div>"

Note that this program invokes another program, pygmentize to actuallyhighlight the source code (pygmentize is part of thePygments project). So, install pygmentize withyour favorite package manager, install Haskell if you have not done soalready, and then compile pygments.hs with:

$ ghc -dynamic pygments.hs

That’s it! Putting it all together, to create a blog post, I can nowdo:

$ pandoc -F pygments -f markdown -t html5 -o blogpost.html blogpost.md

I added some CSS that makes use of the Pygments classes and voilà: youcan now view this blog without having to worry about a JavaScriptcryptocurrency miner hijacking your CPU. You’re welcome.

Remove All Files Except a Few in Bash

Sun, 12 Aug 2018 18:50:05 +0200

How can remove most of the files in a directory in Bash?

$ ls -1153390909910_first15339090991_second15339090992_third15339090993_fourth15339090994_fifth15339090995_sixth15339090996_seventh15339090997_eighth15339090998_nineth15339090999_tenth15339091628_do_not_deleterootroot.sql

We want to delete all files that start with a timestamp (seconds sincethe epoch), except the newest file (15339091628_do_not_delete) and thefiles root and root.sql. The easiest way to do this, is enabling theshell option extglob (“extended globbing”), which allows us to usepatterns to include or exclude files of operations:

$ shopt -s extglob$ rm !(*do_not_delete|root*)

The last command will tell Bash to remove all files, except the onesthat match either one of the patterns (everything ending withdo_not_delete and everything starting with root). We delimit thepatterns by using a pipe character |.

Other patterns that are supported by extglob include:

?(pattern-list)      Matches zero or one occurrence of the given patterns\*(pattern-list)      Matches zero or more occurrences of the given patterns+(pattern-list)      Matches one or more occurrences of the given patterns@(pattern-list)      Matches one of the given patterns!(pattern-list)      Matches anything except one of the given patterns

To disable the extended globbing again:

$ shopt -u extglob

References

To read about all the options that extglob gives you, refer to man bash (search for Pathname Expansion). Searching for shopt in thesame manual page will turn up all shell options. To see which shelloptions are currently enables for your shell, type shopt -p at theprompt.

Tomcat7 Maven Plugin Invalid Byte Tag in Constant Pool 19

Tue, 24 Apr 2018 07:11:48 +0200

I use tomcat7-maven-plugin to spin up a Tomcat 7 container where I canrun my web application. When I added dependencies for log4j2 (version2.11.0) to my project, I got the error:

org.apache.tomcat.util.bcel.classfile.ClassFormatException:Invalid byte tag in constant pool: 19

Apparently, log4j2 is a multi-release jar and older versions of Tomcatcan’t handle that. So I needed to upgrade my Tomcat maven plugin.

Solution: Update your Tomcat

But how do you update your Tomcat? Its informationpage shows thatit hasn’t been updated for a while, the latest version being 2.2 whichruns 7.0.47 by default. MavenCentral,on the other hand, shows that the latest version at the moment of thiswriting is 7.0.86. That’s the version we want.

Change your pom.xml in the following way:

<project>  <properties>    <tomcat.version>7.0.86</tomcat.version>  </properties>  <plugins>    <plugin>      <groupId>org.apache.tomcat.maven</groupId>      <artifactId>tomcat7-maven-plugin</artifactId>      <version>2.2</version>      <configuration>        <path>/</path>        <port>7777</port>      </configuration>      <dependencies>        <dependency>          <groupId>org.apache.tomcat.embed</groupId>          <artifactId>tomcat-embed-core</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-util</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-coyote</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-api</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-jdbc</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-dbcp</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-servlet-api</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-jsp-api</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-jasper</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-jasper-el</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-el-api</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-catalina</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-tribes</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-catalina-ha</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-annotations-api</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat</groupId>          <artifactId>tomcat-juli</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat.embed</groupId>          <artifactId>tomcat-embed-logging-juli</artifactId>          <version>${tomcat.version}</version>        </dependency>        <dependency>          <groupId>org.apache.tomcat.embed</groupId>          <artifactId>tomcat-embed-logging-log4j</artifactId>          <version>${tomcat.version}</version>        </dependency>      </dependencies>    </plugin>  </plugins></project>

How to Write a Custom Appender in Log4j2

Sat, 21 Apr 2018 13:40:59 +0200

Ever wanted to test whether a log statement is triggered? Or whether the formatis the way you want? In this post, we’re going to create a custom log appenderso we can be sure logging is behaving the way we expect.

/* package declaration, imports... */@Plugin(name = "CustomListAppender",        category = Core.CATEGORY_NAME,        elementType = Appender.ELEMENT_TYPE,        printObject = true)public final class CustomListAppender extends AbstractAppender {    // for storing the log events    private List<LogEvent> events = new ArrayList<>();    protected CustomListAppender(            String name,            Filter filter,            Layout<? extends Serializable> layout,            boolean ignoreExceptions) {        super(name, filter, layout, ignoreExceptions);    }    @Override    public void append(LogEvent event) {        if (event instanceof MutableLogEvent) {            events.add(((MutableLogEvent) event).createMemento());        } else {            events.add(event);        }    }    public List<LogEvent> getEvents() {        return events;    }    @PluginFactory    public static CustomListAppender createAppender(            @PluginAttribute("name") String name,            @PluginElement("Layout") Layout<? extends Serializable> layout,            @PluginElement("Filter") Filter filter) {        if (name == null) {            LOGGER.error("No name provided for TestLoggerAppender");            return null;        }        if (layout == null) layout = PatternLayout.createDefaultLayout();        return new CustomListAppender(name, filter, layout, true);    }}

Our CustomListAppender extends AbstractAppender, because thatimplements a lot of the methods from the Appender interface for usthat we would otherwise have to implement ourselves.

The @Plugin annotation identifies this class a plugin that should bepicked up by the PluginManager:

The name attribute defines the name of the appender that can beused in the configuration.
The category attribute should be "Core", because “Core pluginsare those that are directly represented by an element in aconfiguration file, such as an Appender, Layout, Logger or Filter”(source).And we are creating an appender.
The elementType attribute defines which type of element in theCore category this plugin should be. In our case, "appender".
The printObject attribute defines whether our custom plugin classdefines a useful toString() method. We do, because theAbstractAppender class we’re extending is taking care of that forus.

We implement the Appender#append(LogEvent) method to add each event toour events list. If the LogEvent happens to be mutable, we must takecare to create an immutable copy of the event, otherwise subsequent logevents will overwrite it (we will get a list of, say, three log eventsthat are all referencing the same object). We also add a simple gettermethod to retrieve all log events.

For the PluginManager to create our custom plugin, it needs a way toinstantiate it. log4j2 uses a factory method for that, indicated by theannotation @PluginFactory. An appender contains attributes, such as aname, and other elements, such as layouts and filters. To allow forthese, we use the corresponding annotations @PluginAttribute toindicate that a parameter represents an attribute, and @PluginElementto indicate that a parameter represents an element.

To log errors that might occur during this setup, we can make use of theStatusLogger. This logger is available as LOGGER, and is defined inone of the parents of our custom plugin, AbstractLifeCycle. (The levelof log messages that should be visible can be adjusted in the<Configuration status="warn" ...> element.)

Configuration

Configuration:  packages: com.relentlesscoding.logging.plugins  status: warn  appenders:    Console:      name: STDOUT    CustomListAppender:      name: MyVeryOwnListAppender  Loggers:    logger:      -        name: com.relentlesscoding.logging        level: info        AppenderRef:          ref: MyVeryOwnListAppender    Root:      level: error      AppenderRef:        ref: STDOUT

The packages attribute on the Configuration element indicates thepackage that should be scanned by the PluginManager for custom pluginsduring initialization.

How to use our custom list appender?

private CustomListAppender appender;@Beforepublic void setupLogging() {    LoggerContext context = LoggerContext.getContext(false);    Configuration configuration = context.getConfiguration();    appender = configuration.getAppender("MyVeryOwnListAppender");    appender.getEvents().clear();}

When we run tests now, we are able to see all logged events by callingappender.getEvents(). Before each test, we take care to clear the listof the previous log statements.

Unit Test Log4j2 Log Output

Sat, 21 Apr 2018 11:30:19 +0200

Sometimes you want to test if certain log output gets generated whencertain events happen in your application. Here is how I unit test thatusing log4j2 (version 2.11.0).

Use `LoggerContextRule` to get to your `ListAppender` quickly

If you are using JUnit 4, then the quickest solution would be one thatis used by log4j2itself:

import org.apache.logging.log4j.junit.LoggerContextRule;/* other imports */public class LogEventTest {    private static ListAppender appender;    @ClassRule    public static LoggerContextRule init = new LoggerContextRule("log4j2-test.yaml");    @BeforeClass    public static void setupLogging() {        appender = init.getListAppender("List");    }    @Before    public void clearAppender() {        appender.clear();    }    @Test    public void someMethodShouldLogAnError() {        // setup test and invoke logic        List<LogEvent> logEvents = appender.getEvents();        List<String> errors = logEvents.stream()                .filter(event -> event.getLevel().equals(Level.ERROR))                .map(event -> event.getMessage().getFormattedMessage())                .collect(Collectors.toList());        // we logged at least one event of level error        assertThat(errors.size(), is(greaterThanOrEqualTo(1)));        // log event message should contain "wrong" for example        assertThat(errors, everyItem(containsString("wrong")));    }}

The LoggerContextRule provides methods that come in handy whiletesting. Here we use the getListAppender(...) method to get access toan appender that uses a list to store all log events. Before each test,we clear the list, so we have a clean slate for new log events. The testinvokes the code-under-test, requests the log events from the appenderand filters them so that we only have the error log events left. Then wethat at least one error log message was captured and that it containsthe word “wrong”.

Use the `LoggerContext`

Instead of using the class rule (which lets you conveniently pass it thefile name of the configuration), you could also use the LoggerContext:

@BeforeClasspublic static void setupLogging() {    LoggerContext context = LoggerContext.getContext(false);    Logger logger = context.getLogger("com.relentlesscoding");    appender = (ListAppender) logger.getAppenders().get("List");}

This might be your only option if you are working with JUnit 5 (andeventually you will want to migrate to that). In JUnit 5 we can’t useLoggerContextRule anymore, because @Rules don’t longer exist (theywere replaced with an extension mechanism that works differently andlog4j2 doesn’t provide such an extension currently).

Create a working configuration

To get the examples working, we need to define an appender called “List”and a logger in our log4j2 configuration.

log4j2-test.yaml

Configuration:  status: warn  name: TestConfig  appenders:    Console:      name: STDOUT    List:      name: List  Loggers:    logger:      -        name: com.relentlesscoding        AppenderRef:          ref: List    Root:      level: info      AppenderRef:        ref: STDOUT

This configuration will send log events occurring in the packagecom.relentlesscoding and sub-packages to the appender with the name“List” (which is of type ListAppender). This configuration isdefined in YAML, but you can use XML, JSON or the properties format aswell.

Maven dependencies

To get the LoggerContextRule in JUnit 4 working, you need thefollowing:

<dependency>    <groupId>org.apache.logging.log4j</groupId>    <artifactId>log4j-core</artifactId>    <version>2.11.0</version>    <type>test-jar</type></dependency>

To get the YAML log4j2 configuration working, you need:

<dependency>    <groupId>com.fasterxml.jackson.core</groupId>    <artifactId>jackson-databind</artifactId>    <version>2.9.5</version></dependency><dependency>    <groupId>com.fasterxml.jackson.dataformat</groupId>    <artifactId>jackson-dataformat-yaml</artifactId>    <version>2.9.5</version></dependency>

JMockit Fakes

Fri, 06 Apr 2018 16:38:27 +0200

If you are working with microservices, you may find that you are havinga lot of dependencies on other services. If you want your tests to beautonomous, that is, running in isolation of those dependencies, you caneither use a tool like WireMock, or you can usea mocking framework.

Recently, I came across JMockit, a mockingframework for Java.

To mock external dependencies, you can use so-called “fakes”. Say you use aservice that knows the credentials of the customers of your e-store, and youmake requests to this service by using a driver provided by this service, sayCredentialsHttpService. In your development environment (and Jenkins), youdon’t have access to a running service. A solution to this would be a fakeimplementation of CredentialsHttpService, where we would mock the methods thatwe actually call from our code.

public class CredentialsHttpService {    ...    public Optional<CustomerAccount> getCustomerAccount(String id) {        // does HTTP request to service    }    ...}

In our test code, we can now implement the fake:

public class ServiceTest {    @Tested    Service service;    @Test    void test() {        new MockUp<CredentialsHttpService>() {            final static AtomicInteger counter = new AtomicInteger();            @Mock            Optional<CustomerAccount> getCustomerAccount(String id) {                return Optional.of(CustomerAccount.builder()                                        .accountNumber(counter.incrementAndGet())                                        .build());            }        }        service.doFoo();  // eventually invokes our fake                          // getCustomerAccount() implementation    }}

The class to be faked is the type parameter of the MockUp class. Thefake CredentialsHttpService will only mock the methods that areannotated with @Mock. All other methods of the faked class will havetheir real implementation. Mocked methods are not restricted by accessmodifiers: the method may have a private, protected, package-private orpublic access modifier, as long as the method name and number ofparameters and parameter types are the same.

There is a lot of other things that fakes can do in JMockit, see thedocumentation.

Bash Magic Space

Thu, 30 Nov 2017 07:53:08 +0100

Sometimes, a little feedback is appreciated when writing some of the morecomplex Bash constructs. The “magic space” is one the things that can help usget quick feedback.

What does the “magic space” do?

Given the following:

$ find -wholename '*/path/to/file' -print -quit$ man rm$ rm -fv !-2:2

In the last line, feedback would be appreciated to see if we are indeedgoing to delete the second argument of two commands back. If you setBash’ so-called “magic space”, history expansion will take place rightaway after typing a space after !-2:2:

$ rm -fv '*/path/to/file'

How to enable the magic space?

Put the following in your ~/.inputrc:

$if Bash    Space: magic-space$endif

Start a new session, or use bind -f ~/.inputrc to put the changes ineffect immediately.

Other ways to achieve the same

You could also enable shopt -s histverify, which will perform thehistory expansion and give you another opportunity to modify the commandbefore executing it. This requires you to press Enter, though.

Unit Test Grails GORM Formulas

Sun, 19 Nov 2017 11:02:49 +0100

Let’s take a look at how we can test GORM formulas in Grails.

The code for this post is part of my PomoTimer project and can befound on GitHub.

The domain class

We have a domain class Project that models a project that one isworking on when doing a particular work session. It records the name ofthe project (“Writing a blog post”), a status (“active”, “completed”), acreation time, the total time spent on this project and the user theproject belongs to.

The total time is a derived field: it calculates the time spent on thework sessions that belong to this project. It should only calculate thework sessions that have status “done”. Derived fields can be defined inGrails by so-calledformulas.

package com.relentlesscoding.pomotimerclass Project {    String name    ProjectStatus status    Date creationtime    Integer totaltime    User user    static constraints = {        name blank: false, nullable: false, maxSize: 100, unique: true        status blank: true, nullable: false, display: true        creationtime blank: false, nullable: false        totaltime blank: false, nullable: false, min: 0        user blank: false, nullable: false, display: true    }    static mapping = {        totaltime formula: '(select ifnull(sum(select d.seconds from Duration d where ws.duration_id = d.id), 0) from Work_Session ws where ws.project_id = id and ws.status_id in (select st.id from Work_Session_Status st where st.name = \'done\'))'    }    String toString() { return name }}

Testing the formula

So how do we test a formula? Initially, I tried to write integrationtests (grails create-integration-test <className> that create a classwith the grails.testing.mixin.Integration andgrails.transaction.Rollback annotations), but these suffer from thelimitation that each feature method starts a newtransaction,and formula’s aren’t updated until after the transaction is committed(which is never because the transaction is always rolled back). Thiseffectively makes it impossible for an integration test to check whethera formula does what it is supposed to do.

The solution is to write a test specification that extendsHibernateSpec, which allows us to fully use Hibernate in our unittests.

package com.relentlesscoding.pomotimerimport grails.test.hibernate.HibernateSpecclass ProjectTotaltimeSpec extends HibernateSpec {    def 'adding a work session should increase total time'() {        given: 'a new project'        def projectStatus = new ProjectStatus(name: 'foo', description: 'bar')        def user = new User(firstName: 'foo',                            lastName: 'bar',                            userName: 'baz',                            email: 'q@w.com',                            password: 'b' * 10)        def project = new Project(name: 'a new project',                                  status: projectStatus,                                  creationtime: new Date(),                                  totaltime: 0,                                  user: user)        expect: 'total time of the new project is 0 seconds'        project.totaltime == 0        when: 'adding a completed work session to the new project'        def duration = new Duration(seconds: 987)        def done = new WorkSessionStatus(name: 'done', description: 'done')        new WorkSession(starttime: new Date(),                        duration: duration,                        project: project,                        status: done                .save(failOnError: true, flush: true)        // NOTE: a refresh is required to update the domain object        project.refresh()        then: 'total time is equal to duration of completed work session'        project.totaltime == 987    }}

Interesting links

Import Contacts Vcards Into Nextcloud

Sun, 15 Oct 2017 10:37:13 +0200

In this post, we’re going to export our contacts fromGoogle in vCard version 3 format, split thecontacts file and use cadaver to upload all files individually to your addressbook.

The struggle

Last week, I did a fresh install of LingeageOS 14.1 on my OnePlusX and decided not to installany GApps. I have been slowly moving away from using Google servicesand, having found replacements in the form of open-source apps or webinterfaces, I felt confident I would be able to use my phone without aGoogle Play Store or Play Services. (F-Droid is now my sole source ofapps.)

To tackle the problem of storing contacts and a calendar that could besynced, I installed aNextcloud instance on aRaspberry Pi 3. Having installedDAVdroid, I got my phone to synccontacts with Nextcloud, but not all of them: it would stopsynchronizing after some 120 contacts, while I had more than 400.

I decided to try a different approach, so I exported the contacts on myphone in vCard format and triedto upload them to Nextcloud using the aptly named application “contacts”for this. However, this also failed unexpectedly. I’m using Nextcloudversion 12.0.3 and version 2.0.1 of the contact app, but it refuses toaccept vCard version 2.1 (HTTP response code 415: Unsupported mediatype). This, naturally, is the version Android 6 uses to exportcontacts.

After some searching, I found out that if you go tocontacts.google.com, you can downloadyour contacts in vCards version 3. Problem fixed? Well, not so fast:importing 400+ contacts into Nextcloud using the web interface on aRaspberry Pi 3 with an SD card for storage will take a long time. Infact, it never finished over the course of a couple of hours (!), so Ineeded yet another approach.

Fortunately, you can approach your Nextcloud instance through the WebDAVprotocol using tools such ascadaver:

$ cadaver https://192.168.1.14/nextcloud/remote.php/dav

Storing your credentials in a .netrc file in your home directory willenable cadaver to verify your identity without prompting, making itsuitable for scripting:

machine 192.168.1.14login foopassword correcthorsebatterystaple

cadaver allows you to traverse the directories of the remote filesystem over WebDAV. To put a single local contacts file (from yourworking machine) to the remote Raspberry Pi, you could tell it to:

dav:/nextcloud/remote.php/dav/> cd addressbooks/users/{username}/{addressbookname}dav:/nextcloud/remote.php/dav/addressbooks/users/foo/Contacts/> put /home/foo/all.vcf all.vcf

I had a single vcf file with 400+ contacts in them, but afteruploading it this way, only a single contact was being displayed.Apparently, the Nextcloud’s contacts app assumes a single vcf filecontains only a single contact. New challenge: we need to split thissingle vcf file containing multiple contacts into separate files thatwe can then upload to Nextcloud.

To split the contacts, we can use awk:

BEGIN {    RS="END:VCARD\r?\n"    FS="\n"}{    command = "echo -n $(pwgen 20 1).vcf"    command | getline filename    close(command)    print $0 "END:VCARD" > filename}

This separates the contacts on the record separator END:VCARD andgenerates a random filename to store the individual contact in. (I alsowrote a Java program to do the samething, which is faster whensplitting large files).

Obviously, it would be convenient now if we could upload all these filesin one go. cadaver does provides the mput action to do so, but I didnot get it to work with wildcards. So instead, I created a file withput commands:

for file in *.vcf; do    echo "put $(pwd)/$file addressbooks/users/foo/Contacts/$file" >> commandsdone

And then provided this as input to cadaver:

$ cadaver http://192.168.1.14/nextcloud/remote.php/dav <<< $(cat commands)

This may take a while (it took around an hour for 400+ contacts), but atleast you get to see the progress as each request is made and processed.And voilà, all the contacts are displayed correctly in Nextcloud.

Always on Vpn and Captive Portals

Tue, 12 Sep 2017 20:48:10 +0200

I was attending a meetup that was taking place in a bar in Utrecht. Thefirst thing you want to do is to make a connection to the internet andget started. The location used a captiveportal, however. Youknow: you have the name of the wireless network (SSID) and the password,but when you try to open any web page, you are directed towards a loginpage where you have to accept the terms and conditions of whoever isoperating the network.

But what if you use an always-on VPN? You cannot connect to the network,because your MAC and IP address are not whitelisted yet by the operator.And you cannot get to the login page, because you do not allow anytraffic outside your VPN.

UFW

I use ufw (uncomplicated firewall) as my firewall of choice, mainlybecause the alternative, iptables, always looked too complicated andufw served its purpose. The rules I have for ufw are currently:

# ufw status verboseStatus: activeLogging: on (low)Default: deny (incoming), deny (outgoing), disabled (routed)New profiles: skipTo                         Action      From--                         ------      ----Anywhere on wlp1s0         ALLOW IN    192.168.178.0/24Anywhere                   ALLOW OUT   Anywhere on tun0-unrooted192.168.178.0/24           ALLOW OUT   Anywhere on wlp1s01194                       ALLOW OUT   Anywhere on wlp1s0Anywhere (v6)              ALLOW OUT   Anywhere (v6) on tun0-unrooted1194 (v6)                  ALLOW OUT   Anywhere (v6) on wlp1s0

By default, I deny all incoming and outgoing connections. I only allowincoming connections from hosts on the same network. As for outgoingconnections, I only allow them to other hosts over the LAN (to anyport), and everywhere else only over port 1194 (the OpenVPN port).(wlp1s0 is the name of my wireless interface, tun0-unrooted of theVPN tunnel. These rules were inspired by this Arch Linuxarticle).

So we need to allow some traffic outside of the VPN tunnel to accept theterms and conditions and to register our machine at the captive portal.The best thing to do would be to allow a single, trusted application toaccess this portal, one that would be used exclusively for this task. Ifyou would allow you regular browser to bypass the VPN, it would send allkind of traffic over the untrusted network for the rest of the world tofreely sniff around in (think add-ons, other browser tabs, automaticupdates). So we would need a dedicated web browser for this task. I’m onLinux using Firefox as my default browser, so GNOMEWeb would be a good choice forthis purpose. (Gnome Web was previously known as Epiphany, and is stillavailable under that name on a lot of distributions) .

First, we need to determine what kind of traffic we want to allow. Theapplication will need to have outbound access to ports 80 (HTTP) and 443(HTTPS) for web traffic, and it will also need to be able to resolvedomain names using DNS, so port 53 should also be opened.

UFW Profiles

However, it’s not that easy to allow one particular application accessto the internet if you use UFW. When you look at the man page for UFW,you see you can specify “apps”. Apps (or application profiles) arebasically just text files in INI-format that live in the/etc/ufw/applications.d/ folder.

To list all (predefined) profiles:

# ufw app list

To create a profile for our purposes, we put the following in a filecalled ufw-webbrowser:

[Epiphany]title=Epiphanydescription=Epiphany web browserports=80/tcp|443/tcp|53

The “ports” field is clarified in the man page:

The ‘ports’ field may specify a ‘|’-separated list of ports/protocolswhere the protocol is optional. A comma-separated list or a range(specified with ‘start:end’) may also be used to specify multipleports, in which case the protocol is required.

In our case we allow TCP traffic over ports 80 (HTTP) and 443 (HTTPS)and both UDP and TCP traffic over 53 (DNS). We can now use this profile:

# ufw insert 1 allow out to any app EpiphanyRule addedRule added (v6)# ufw status verbose[...snip...]To                         Action      From--                         ------      ----Anywhere on wlp1s0         ALLOW IN    192.168.178.0/2480/tcp (Epiphany)          ALLOW OUT   Anywhere443/tcp (Epiphany)         ALLOW OUT   Anywhere53 (Epiphany)              ALLOW OUT   AnywhereAnywhere                   ALLOW OUT   Anywhere on tun0-unrooted192.168.178.0/24           ALLOW OUT   Anywhere on wlp1s01194                       ALLOW OUT   Anywhere on wlp1s080/tcp (Epiphany (v6))     ALLOW OUT   Anywhere (v6)443/tcp (Epiphany (v6))    ALLOW OUT   Anywhere (v6)53 (Epiphany (v6))         ALLOW OUT   Anywhere (v6)Anywhere (v6)              ALLOW OUT   Anywhere (v6) on tun0-unrooted1194 (v6)                  ALLOW OUT   Anywhere (v6) on wlp1s0# ufw reload  # don't forget to reload firewall after making changes!Firewall reloaded

(Note that we use insert 1 to make sure the rule is placed in thefirst position. With UFW, the first rule matched wins.)

Now we can use our dedicated browser to go to the captive portal pageand accept the terms and conditions.

Checking traffic with `wireshark`

You need to be careful, however, not to use any other applicationsduring this time. If you launch Firefox, for example, it can also usethe opened ports to communicate with the outside world. I’d like to useWireshark to see what communications are taking place during this time.

When you are registered with the WiFi provider and are done with thecaptive portal, you should first disable the profile again with UFW. Wecan do this by specifying the rule we added earlier, but prependingdelete:

# ufw delete allow out to any app Epiphany

Another way to delete rules from UFW is by first doing ufw status numbered and then ufw delete <number>. However, since we have added 6rules, this may take a while. Also, if you can’t remember the exact rulethat was used, you can use ufw show added to show all added rules andtheir syntax.

Better solutions: beyond UFW

Now, we see that using UFW isn’t exactly ideal to deal with always-onVPN and captive portals. What if you have an email application (orsomething else) running in the background when you have allowed allthose ports to bypass the VPN tunnel? And also, you have to enable anddisable the application profile every time you encounter a captiveportal. It would be better if we could allow only a single, named,demarcated application to bypass the VPN.

One solution I’ve read about makes use of what I like to call “theAndroid way”: every installed application is a user with its own homedirectory. This means that applications don’t have access to each otherfiles, but more importantly, this gives the opportunity to allow only aspecific user to access the internet outside of the VPN. This way, wecould create a user epiphany that runs Gnome Web to access the captiveportal.

AFWall+, an open-source Androidapplication, uses this method to implement a pretty effective firewall.It also uses iptables as a back-end. I might have to finally bite thebullet and learn iptables after all…

Tutorial Rapid GUI Development With Qt Designer and PyQt

Fri, 25 Aug 2017 10:02:43 +0200

Confession: I am the opposite of the lazy coder. I like doing things thehard way. Whether it’s developing Java in Vim without code completion,running JUnit tests on the command line (don’t forget to specify all 42dependencies in the colon-separated classpath!), creating a LaTeX graphthat looks “just right”, or writing sqlplus scripts instead of usingSQL Developer (GUIs are for amateurs), I always assumed that doing sowould make me a better programmer.

So when I was just a fledgling programmer, and I had to design andcreate some dialog windows for a side project of mine (an awesomeadd-on toAnkiSRS written in Python and Qt), I didwhat I always do: find a good resource to learnPyQt and then code everything by hand.Now, since Python is a dynamic language and I used to develop thisadd-on in Vim, I had no code completion whatsoever, and only the Qtdocumentation and that tutorial to go by. Making things even moreinteresting, is that the documentation on Qt is in C++, and is full ofstuff like this:

Obviously, I had no idea what all the asterisks and ampersands meant andhad to use good-old trial and error to see what would stick in Python.Now, on the plus side, I experimented quite a lot with the code, but nothaving code completion makes it really hard to learn the API. (And evennow, using PyCharm, code completion will not always work, because ofPython being a dynamically-typed language and all. I am still looking atthe Qt documentation quite a bit.)

One thing I noticed, though, is that the guy who develops Anki, DamienElmes, had all these .ui files lying around,and a bunch more files that read: “WARNING! All changes made to thisfile will be lost!”. Ugh, generated code! None of the dedicated,soul-cleansing and honest hard work that will shape you as a softwaredeveloper. It goes without saying I stayed far away from that kind oflaziness.

It took me some time to come around on this. One day, I actually got ajob as a professional software developer and I had much less time towork on my side-projects. So, when I wanted to implement another featurefor my Anki add-on, I found I had too little time to do it theold-fashioned way, and decided to give Qt Designer a go. Surprisingly, Ifound out it can actually help you tremendously to learn how Qt works(and also save you a bunch of time). Qt Designer allows you to visuallycreate windows using a drag-and-drop interface, then spews out an XMLrepresentation of that GUI, which can be converted to code. Thatgenerated code will show you possibilities that would have taken a longtime and a lot of StackOverflowing to figure out on your own.

So, to help other people discover the wonders of this program, here is alittle tutorial on how to do RAD and how to get a dialog window up andrunning with Qt Designer 4 and Python 3.

Installation

First, we need to install Qt Designer. On Arch Linux, it’s part of theqt4 package and you’ll also need python-pyqt4 for the Pythonbindings. (On Ubuntu, you have to install both qt4-designer and thepython-qt4 packages.)

Our goal

Our goal is to create a simple dialog window that has a text input fieldwhere we can type our name, and have a label that will display “Hellothere, $userName”. Yes, things will be that exciting. Along the way,we will learn how to assign emitted signals to slots and how to handleevents.

Creating a dialog

Fire up Qt Designer, and you will be presented with a “New form” dialog(if you do not see it, go to File > New…).

For this tutorial, we are going to choose a fairly small “Dialog withButtons Bottom”:

To the left are the widgets that we can add to our freshly createddialog, to the right, from top to bottom, we see the currently addedwidgets, the properties of those widgets and the signals and slotscurrently assigned to the dialog window.

I’m going to add a text label and a so-called line editor to our widget.To make sure they will align nicely, I will put them together in ahorizontal container, a QHBoxLayout:

In the object inspector, we can see the hierarchy of added widgets:

This is all simple drag-and-drop: we select a widget from the left pane,drag it to the desired location in the dialog and release the mouse.

Finally, we add two more label: one at the top, instructing the userwhat to do, and a second one near the bottom, where we soon will displayour message.

Qt Designer provides an easy way to connect signals to slots. If you goto Edit > Edit Signals/Slots (or press F4) you will be presented witha graphical overview of the currently assigned signals and slots. Whenwe start out, the button box at the bottom already emits two signals:rejected and accepted, from the Cancel and Ok button respectively:

The signals rejected() and accepted() are emitted when the canceland OK buttons are clicked respectively, and the ground symbols indicatethe object that is interested in these signals: in this case the dialogwindow itself. Signals are handled by slots, and so the QDialog willneed to have slots for these signals. The slots (or handlers) are namedreject() and accept() in this instance, and since they are defaultslots provided by Qt, they already exist in QDialog, so we won’t haveto do anything (except if we want to override their default behavior).

What we want to do now is “catch” the textEdited signal from the lineeditor widget and create a slot in the dialog window that will handleit. This new slot we’ll call say_hello. So we click the line editorwidget and drag a line to anywhere on the dialog window: a ground symbolshould be visible:

In the window that appears now, we can select the signal that we areinterested in (textEdited) and assign it to a predefined slot, or wecan click on Edit… and create our own slot. Let’s do that:

We click the green plus sign, and type in the name of our new slot(say_hello):

The result will now look like:

In Qt Designer, you can preview your creation by going to Form >Preview… (or pressing Ctrl + R). Notice, however, that typing text inthe line editor won’t do anything yet. This is because we haven’twritten any implementation code for it. In the next section we will seehow to do that.

You can also see the code that will be generated by going to Form >View code…, although this code is going to be in C++.

Okay, enough with designing our dialog window, let’s get to actualPython coding.

Generating code (or not)

When we save our project in Qt Designer, it will create a .ui file,which is just XML containing all the properties (widgets, sizes, signals& slots, etc.) that make up the GUI.

PyQt comes with a program, pyuic4, that can convert these .ui filesto Python code. Two interesting command-line options are --preview (or-p for short), that will allow you to preview the dynamically createdGUI, and --execute (or -x for short), that will generate Python codethat can be executed as a stand-alone. The --output switch (or -o)allows you to specify a filename where the code will be saved.

In our case, creating a stand-alone executable is not going to work,because we have added a custom slot (say_hello) that needs to beimplemented first. So we will use pyuic4 to generate the form class:

$ pyuic4 relentless_dialog.ui -o relentless_dialog.py

This is the result:

# -*- coding: utf-8 -*-# Form implementation generated from reading ui file 'relentless_dialog.ui'## Created by: PyQt4 UI code generator 4.12.1## WARNING! All changes made in this file will be lost!from PyQt4 import QtCore, QtGuitry:    _fromUtf8 = QtCore.QString.fromUtf8except AttributeError:    def _fromUtf8(s):        return stry:    _encoding = QtGui.QApplication.UnicodeUTF8    def _translate(context, text, disambig):        return QtGui.QApplication.translate(                context, text, disambig, _encoding)except AttributeError:    def _translate(context, text, disambig):        return QtGui.QApplication.translate(context, text, disambig)class Ui_Dialog(object):    def setupUi(self, Dialog):        Dialog.setObjectName(_fromUtf8("Dialog"))        Dialog.resize(250, 320)        self.buttonBox = QtGui.QDialogButtonBox(Dialog)        self.buttonBox.setGeometry(QtCore.QRect(10, 270, 221, 41))        self.buttonBox.setOrientation(QtCore.Qt.Horizontal)        self.buttonBox.setStandardButtons(                QtGui.QDialogButtonBox.Cancel|QtGui.QDialogButtonBox.Ok)        self.buttonBox.setObjectName(_fromUtf8("buttonBox"))        self.horizontalLayoutWidget = QtGui.QWidget(Dialog)        self.horizontalLayoutWidget.setGeometry(                QtCore.QRect(10, 40, 231, 80))        self.horizontalLayoutWidget.setObjectName(                _fromUtf8("horizontalLayoutWidget"))        self.horizontalLayout = QtGui.QHBoxLayout(                self.horizontalLayoutWidget)        self.horizontalLayout.setMargin(0)        self.horizontalLayout.setObjectName(_fromUtf8("horizontalLayout"))        self.label = QtGui.QLabel(self.horizontalLayoutWidget)        self.label.setObjectName(_fromUtf8("label"))        self.horizontalLayout.addWidget(self.label)        self.lineEdit = QtGui.QLineEdit(self.horizontalLayoutWidget)        self.lineEdit.setObjectName(_fromUtf8("lineEdit"))        self.horizontalLayout.addWidget(self.lineEdit)        self.label_2 = QtGui.QLabel(Dialog)        self.label_2.setGeometry(QtCore.QRect(0, 10, 251, 20))        self.label_2.setObjectName(_fromUtf8("label_2"))        self.label_3 = QtGui.QLabel(Dialog)        self.label_3.setGeometry(QtCore.QRect(2, 190, 241, 20))        self.label_3.setObjectName(_fromUtf8("label_3"))        self.retranslateUi(Dialog)        QtCore.QObject.connect(self.buttonBox,                               QtCore.SIGNAL(_fromUtf8("accepted()")),                               Dialog.accept)        QtCore.QObject.connect(self.buttonBox,                               QtCore.SIGNAL(_fromUtf8("rejected()")),                               Dialog.reject)        QtCore.QObject.connect(                self.lineEdit,                QtCore.SIGNAL(_fromUtf8("textChanged(QString)")),                Dialog.say_hello)        QtCore.QMetaObject.connectSlotsByName(Dialog)    def retranslateUi(self, Dialog):        Dialog.setWindowTitle(_translate("Dialog", "Dialog", None))        self.label.setText(_translate("Dialog", "Name:", None))        self.label_2.setText(_translate(                             "Dialog",                             "<html><head/><body><p align="center">" +                             "Please enter your name</p></body></html>",                             None))        self.label_3.setText(_translate(                             "Dialog",                             "<html><head/><body><p align="center">" +                             "[Where our message will appear]" +                             "</p></body></html>",                             None))

It is here that you can learn a lot about how PyQt works, even thoughsome statements seem a bit baroque, like the binding of thetextChanged signal to our custom slot:

QtCore.QObject.connect(        self.lineEdit,        QtCore.SIGNAL(_fromUtf8("textChanged(QString)")),        Dialog.say_hello)

If you would write this yourself, you would probably prefer:

self.lineEdit.textChanged[str].connect(Dialog.say_hello)

(Incidentally, the text between the square brackets intextChanged[str] indicates that a single argument of this type ispassed to the slot.)

Bringing it all together

Having generated our form class, we can now create a base class thatwill use this class:

import sysfrom PyQt4 import QtGuifrom relentless_dialog import Ui_Dialogclass MyDialog(QtGui.QDialog):    def __init__(self):        super(MyDialog, self).__init__()        self.ui = Ui_Dialog()        self.ui.setupUi(self)        self.show()    def say_hello(self, user_text):        text = "Hello there, {0}!".format(user_text)        self.ui.label_3.setText(text)def main():    app = QtGui.QApplication(sys.argv)    dialog = MyDialog()    app.exec_()if __name__ == "__main__":    main()

When we are passing self to self.ui.setupUi, we are passing thewidget (a QDialog) in which the user interface will be created.

We implement our custom slot by defining a method say_hello that takesa single argument (the user-provided text from the line editor). Wecraft a witty sentence with it, and set it as the label text.

As mentioned before, we could also dynamically create the GUI directlyfrom the .ui file:

import sysfrom PyQt4 import QtGuifrom PyQt4 import uicclass MyDialog(QtGui.QDialog):    def __init__(self):        super(MyDialog, self).__init__()        dialog = uic.loadUi("relentless_dialog.ui", self)        dialog.show()# etc

Running the application

Running this will show the following (drum roll):

How to Use Groovy's CliBuilder

Fri, 11 Aug 2017 16:35:57 +0200

Let’s use Groovy’s CliBuilder to create CLI programs that can take flags.

Intro

I write quite some scripts at work, either for myself or for my team. Iused to write these scripts in Bash, but since we are using Windows atwork, it made sense to switch to a more general scripting language.Because we’re a Java shop, Groovy seemed like a natural choice. Now,it’s often very convenient to have named command-line arguments (e.g.tool --output out.ext --input in.ext) as opposed to positionalarguments (e.g. tool out.ext in.ext): named arguments allow forputting the arguments in any order you want, and you can leave outoptional arguments and use the defaults. Groovy’s CliBuilder makesthings especially easy.

A sample program

weather.groovy

@Grab(group='commons-cli', module='commons-cli', version='1.4')class Main {    static main(args) {        CommandLineInterface cli = CommandLineInterface.INSTANCE        cli.parse(args)    }}enum CommandLineInterface {    INSTANCE    CliBuilder cliBuilder    CommandLineInterface() {        cliBuilder = new CliBuilder(                usage: 'weather [<options>]',                header: 'Options:',                footer: 'And here we put footer text.'        )        // set the amount of columns the usage message will be wide        cliBuilder.width = 80  // default is 74        cliBuilder.with {            h longOpt: 'help', 'Print this help text and exit.'            n(longOpt: 'max-count', args: 1, argName: 'number',                    'Limit the number of days shown')            '1' longOpt: 'one', "Show today's forecast"            '2' longOpt: 'two', "Show today's and tomorrow's forecast"            _(longOpt: 'url', args: 1, argName: 'URL',                    'Use given URL to query for weather.')            D(args: 2, valueSeparator: '=', argName: 'property=value',                    'Use value for given property.')        }    }    void parse(args) {        OptionAccessor options = cliBuilder.parse(args)        if (!options) {            System.err << 'Error while parsing command-line options.\n'            System.exit 1        }        if (options.h) {            cliBuilder.usage()            System.exit 0        }        if (options.n) {            // handle user's input        }        if (options.'1') {            // show weather for one day        }        if (options.url) {            URI uri = new URI(options.url)        }        if (options.Ds) {            assert options.Ds.class == ArrayList.class        }    }}

When invoked with groovy weather.groovy --help, the program willoutput:

usage: weather []Options: -1,--one           Show today's forecast -2,--two           Show today's and tomorrow's forecast -D                 Use value for given property. -h,--help          Print this help text and exit. -n,--max-count     Limit the number of days shown    --url           Use given URL to query for weather.And here can put footer text.

You can specify short and long names: cliBuilder.n specifies that theprogram will accept an argument -n. To also specify a long name, weadd longOpt: max-count. To use only a long name, we can replace theshort name by _: cliBuilder._(longName: 'wow-such-long'). Note thatcliBuilder._ can be reused several times (in contrast to other shortnames).

The number of arguments can be indicated by using the args parameter.If you use more than one parameter, a valueSeparator must beindicated, which is used to split up the argument. I found that thenumber assigned to args is not very strict: when called with argument-Dmyprop, the following is valid.

assert options.Ds == ['myprop']

Note how we append an s to the option name (options.Ds) to get alist of properties and values. A normal invocation groovy weather.groovy -Dpancakes=yesplease would result in:

assert options.Ds == ['pancakes', 'yesplease']

When required: true is set, the program will shout at you if theargument is not used. For example, if we specify cliBuilder.q(required: true) but fail to provide the -q argument on the command line, theprogram will exit with error: Missing required option: q.

One thing to remember is that, if you want to use digits as argumentnames, you have to stringify them first by putting quotation marksaround them: cliBuilder.'1'.

More on CliBuilder

Check the Groovy documentation onCliBuilderfor more information.

Dependencies

Groovy’s CliBuilder depends on cli-commons:

build.gradle

dependencies {    compile group: 'commons-cli', name: 'commons-cli', version: '1.4'}

Using Groovy's AntBuilder to Zip and Unzip Files

Fri, 04 Aug 2017 12:52:20 +0200

Need to zip or unzip files? Let’s take a look at how Groovy solves this.

Suppose we need to zip a bunch of Python files (without their compiledcounterparts *.pyc). The next (admittedly contrived) example shows howwe could go about doing something like this in Groovy:

File srcDir = new File('/home/neftas/Downloads/scripts/')File destDir = new File(srcDir, 'antexample')File zippedFile = new File(srcDir, 'antzipped.zip')def allPythonFilenames = srcDir.list()    .sort()    .findAll { it.endsWith('.py') }assert ['a.py', 'b.py', 'c.py'] == allPythonFilenamesdef ant = new AntBuilder()ant.with {    echo 'begin zipping and unzipping'    zip(destfile: zippedFile, basedir: srcDir, includes: '*.py')    mkdir(dir: destDir)    unzip(src: zippedFile, dest: destDir)    echo 'done zipping and unzipping'}// zip file is createdassert zippedFile.exists()// contents of zip file should match content of source directorydef commands = ['bash', '-c', "zipinfo -1 ${zippedFile.name}"]def process = commands.execute(null, srcDir)def contentsOfZip = process.text.split(System.lineSeparator)assert contentsOfZip.sort() == allPythonFilenames// all files should be unpacked from the zip into the right directoryassert destDir.list().sort() == allPythonFilenamesant.with {    echo 'deleting created files'    // notice nested Ant task    delete(includeEmptyDirs: true) {        fileset(dir: destDir)        fileset(file: zippedFile)    }    echo 'deleted created files'}

We can use all the power of Ant in our Groovy scripts. Ant task namesmap to Groovy methods (see the zip method above) and the attributesare passed as maps to these methods. What’s even better, there is noneed to stringify the arguments of these attributes (as is required inAnt’s XML), but we can directly use the correct datatypes (e.g. in thekey-value pair destfile: zippedFile, zippedFile is of typejava.util.File). All attributes of Ant tasks are available to use withAntBuilder (see Ant manual for ziptask).

To use nested Ant tasks, we create closures, so that Ant’s:

<delete includeEmptyDirs="true">    <fileset dir="lib">    <fileset file="file.ext"></delete>

results in the following Groovy code:

ant.delete(includeEmptyDirs: true) {    fileset(dir: 'lib')    fileset(file: 'file.ext')}

Invoking from Gradle

AntBuilder is also available in Gradle. This will come in handy whenyou don’t know the “Gradle way” of doing something, but do know how totackle it in Ant.

Dependencies

Groovy (and Gradle) come with a bundled version of AntBuilder, so youshould be good shape when creating a Groovy script, but if you want touse AntBuilder in a project, you should add ant and ant-launcherto your build.gradle:

dependencies {    compile group: 'ant', name: 'ant',  version: '1.7.0'    compile group: 'ant', name: 'ant-launcher',  version: '1.6.5'}

I Like My Method Pointers With Curry in Groovy

Fri, 28 Jul 2017 17:13:09 +0200

Invoking a method several times can easily be done in Java in a loop.But what if we have several methods that need to be executed severaltimes? Since Java does not feature pointers to functions, we cannot passfunctions around, and cannot create a generic method that takes apointer to a method and executes it. This is where Groovy shines.

Suppose we want to check on the status of some external system or work.We want to check the status every half a second for a maximum of tentimes in total.

// the generic type in Closure<Boolean> indicates its return typeboolean isOperationSuccessful(int numOfTries, Closure<Boolean> action) {    Thread.sleep(500)    for (i in 0..<numOfTries) {        if (action()) {            return true        } else {            Thread.sleep(500)        }    }}// we can pass closures...def check = { int n -> new Random().nextInt(10) == n }println isOperationSuccessful(10, check.curry(3))// ... as well as methodsboolean isWorkDone(int id) {    // check if particular external system identified by `id` is done    return new Random().nextInt(2) == 1}// `.&` is the method pointer operatorprintln isOperationSuccessful(5, this.&isWorkDone.curry(1))

We can pass methods around by referencing them with the method pointeroperator.&:

def pointerToUpper = 'hello world'.&toUpperCaseassert pointerToUpper() == 'HELLO WORLD'

By using Groovy’s take oncurrying,we can fix parameters of a closure (or method):

def power(double n, int exponent) { n ** exponent }def squared = this.&power.rcurry(2)assert squared(12) == 144

rcurry fixes the right-most argument of the method, curry theleft-most. (And yes, there is also an index-based method: ncurry).Both return a new Closure that takes one less argument.

How to Ignore an Invalid SSL Certificate in Java

Mon, 24 Jul 2017 07:29:34 +0200

Sometimes during development it is useful to use a certificate whose CN(Common Name) does not match the host name in the URL, for examplelocalhost. In these cases Java will throw an SSLHandshakeException.How can we easily disable certificate checking for localhost and otherdomains of our choosing?

Your own certificate checking

The easiest way is to create your own class that implements theinterface HostnameVerifier:

WhitelistHostnameVerifier.java

package com.relentlesscoding.https;import javax.net.ssl.HostnameVerifier;import javax.net.ssl.HttpsURLConnection;import javax.net.ssl.SSLSession;import java.util.HashSet;import java.util.Set;// singletonenum WhitelistHostnameVerifier implements HostnameVerifier {    // these hosts get whitelisted    INSTANCE("localhost", "sub.domain.com");    private Set whitelist = new HashSet<>();    private HostnameVerifier defaultHostnameVerifier =            HttpsURLConnection.getDefaultHostnameVerifier();    WhitelistHostnameVerifier(String... hostnames) {        for (String hostname : hostnames) {            whitelist.add(hostname);        }    }    @Override    public boolean verify(String host, SSLSession session) {        if (whitelist.contains(host)) {            return true;        }        // important: use default verifier for all other hosts        return defaultHostnameVerifier.verify(host, session);    }}

Main.java

// use our HostnameVerifier for all future connectionsHttpsURLConnection.setDefaultHostnameVerifier(        WhitelistHostnameVerifier.INSTANCE);final String url = "https://relentlesscoding.com";HttpsURLConnection conn =        (HttpsURLConnection) new URL(url).openConnection();System.out.println(conn.getResponseCode());

HttpsURLConnection.DefaultHostnameVerifier#verify always returnsfalse, so we might as well return false ourselves, but this way thecode will keep working when the implementation of HttpsURLConnectionchanges.

Note: you can set the default HostnameVerifier globally by using thestatic method setDefaultHostnameVerifier, or you can set it perinstance, using conn.setHostnameVerifier.

Problems with Let’s Encrypt certificates

One thing I found out while testing this, was that OpenJDK version1.8.0_131 won’t correctly handle Let’s Encrypt certificates:

java.io.IOException: HTTPS hostname wrong: should be <relentlesscoding.com>

This is due to the fact that Let’s Encrypt certificates are not presentin the Java installation. Upgrading to version 1.8.0_141 resolves this(for Oracle versions >= 8U101 or >= 7U111).

Whitelisting, NOT disabling certificate checking

The best approach here is whitelisting. This will keep the certificatechecking in place for most sites, and will only disable it forpre-approved hosts. I saw quite a bit of examples online that disablecertificate checking completely, by writing the verify method asfollows:

public boolean verify(String hostname, SSLSession session) {    return true;}

This is not secure, as it completely ignores all invalidcertificates.

Creating Custom Intershop ISML Functions With CustomTag

Sat, 22 Jul 2017 17:27:33 +0200

In this post, I’ll look at leveraging Intershop’s CustomTag to create customfunctionality in ISML templates.

The problem

I wanted to capitalize a certain value from the pipeline dictionary inan ISML template. Intershop provides ISML functions of the kind ucaseand lcase to transform a string to uppercase or lowercaserespectively. When the value of the dictionary name Product:Name isAwesomeNotebook:

<isprint value="#ucase(Product:Name)#">

This will print AWESOMENOTEBOOK. Unfortunately, a similar function tocapitalize a string is not provided by Intershop. Ideally, you wouldwant to be able to implement your own methods, so that you could type:

<isprint value="#capitalize(Product:Name)#">

What about custom modules?

Intershop does provide the ability to create custom modules. These arebasically just tags you can give whatever name you like, and you canprovide an implementation in Java or ISML. The interface is defined in amodules.isml file. You could, for example, define a custom tagisprintdefault that prints a default value whenever the value passedto it is null or the empty string:

<isprintdefault default="(unknown)" value="#Product:Price">

This would print (unknown) every time Product:Price does not containa value.

This solution, however, has two drawbacks:

For a Java implementation, this would come down to writingscriptlet. A few lines of scriptlet is fine ofcourse, but as thesize of the code increases, it becomes harder and harder tomaintain, because there is no compilation-time check and debuggingis nearly impossible (say hello to a thousand print statements).
Also, it is cumbersome for implementing something like thecapitalize function mentioned above, because you would have toprovide the input to the custom module, put the result in thepipeline dictionary, and then use the pipeline dictionary name:
```
<iscapitalize dictname="capitalizedStr" input="#Product:Name#"><isprint value="#capitalizedStr#">
```
Instead of just using the returned capitalized string directly:
```
<isprint value="#capitalize(Product:Name)#">
```

Solution: ISML Custom Tag

An ISML customtag looksjust like a custom module:

<ISUUID name="CategoryRenderEntityID">

(Incidently, this creates a UUID and stores it in the pipelinedictionary variable name CategoryRenderEntityID). According to thedocumentation, the difference between a custom tag and a custommodule is that the former “provides a significant performance increase”compared to the latter.

Two purposes

Custom tags can be used for two purposes:

Performing a single operation, like we saw with the ISUUID customtag above, where the tag performs a single action (creating a newUUID and putting it in the pipeline dictionary).
Inserting an instance of a helper or utilities class into thepipeline dictionary.

Creating an ISML Custom Tag

Say, we want to create an ISML Custom Tag named <ishelper>. First, wewould define it in a modules.properties file, located at<cartridge>/staticfiles/cartridge/config/modules.properties:

ismodules.helper.class=custom_base.internal.modules.Helperismodules.helper.isStrict=falseismodules.helper.parameters=aliasismodules.helper.parameter.alias.type=java.lang.Stringismodules.helper.parameter.alias.isRequired=false

The class property should point to the implementing Java class. Sincea pipeline dictionary is available to custom tags, we can define whetherwe want a clean, strict dictionary or not, much like we can do inpipelines. Parameters and return values can be defined by providingtheir name, type and whether or not they are mandatory.

So, the properties file above defines a custom tag <ishelper> (put aleading is to your lowercase class name), that has an optionalattribute alias of type String. Other parameters can be added byusing comma’s:

ismodules.helper.parameters=alias,more,params

Don’t forget to also add a type and whether the parameter is required ornot. The type of the parameter can be any Java class, so you’re notlimited to just strings.

The Java class with the implementing code must extend the abstract classCustomTag. It is required to override the method processOpenTag,which will be called every time the custom tag is encountered in theISML template.

Now then, we can create “custom ISML functions” by adding an instance ofour Helper class to the pipeline dictionary:

public class Helper extends CustomTag {    @Override    public void processOpenTag(PageContext paramPageContext,                                ServletResponse paramServletResponse,                                AbstractTemplate paramAbstractTemplate,                                int paramInt)            throws IOException, ServletException    {        PipelineDictionary dict = AbstractTemplate                .getTemplateExecutionConfig()                .getPipelineDictionary();        String alias = dict.getOptional("alias", "Helper");        dict.put(alias, new Helper());    }}

If we wanted a function capitalize in our templates, we could add sucha method to our Java class:

public String getCapitalize(String str) {    /* do groundbreaking stuff */    return capitalizedStr;}

Note that the method name should start with get in order for it to beavailable in the pipeline dictionary in our templates. We can now usethe method as follows:

<isprint value="#Helper:Capitalize(Product:Name)#">

Don’t forget, however, to add the custom tag somewhere at the beginningof the template before using it:

<ishelper alias="MyCustomUtilsClass"><!-- snip --><isloop iterator="MyProducts" alias="product">    <isprint value="#MyCustomUtilsClass:Capitalize(product:Name)#"</isloop>

(Note that the alias attribute is optional. If you leave it out, itwill default to Helper: see the implementation above. This will be thealias of the class we’re using to invoke methods on:

<isprint value="#MyCustomUtilsClass:noArgMethod()#">

Remember that the name of the actual class that contains the custom tagis part of the tag name: ishelper.)

Alternative method of making a Java class available in the pipeline dictionary

Obviously, we do not have to rely on an ISML custom tag to put aninstance of a Java class in the pipeline dictionary. We could justcreate a pipelet that does this, and put this pipelet in a pipeline thatgets called a lot (a Prefix pipeline would be good). This would removethe need to put <ishelper> (or whatever we named the custom tag) atthe top of our templates. But the helper class would only be availableif the Prefix pipeline was called, so you would need to check thatcarefully. Putting the custom tag at the top of the template makes theimport explicit and easily verifiable.

A word of caution

Custom tags are very handy, but they extend the internal CustomTagclass. This means Intershop does not encourage using them in customproject development. In fact, you won’t find any information about themin the documentation.

Java Stateful Sessions or How to Properly Send Cookies With Each Redirect Request

Fri, 14 Jul 2017 19:05:45 +0200

Sometimes you need to log in to some webpage programmatically. I ran into one ofthose pages where, when the login succeeds, you’re being redirected to anotherpage, and then to another (something along the lines of ‘Please login’ -> ‘Youare successfully logged in’ -> ‘Admin panel’). So I needed to write somethingthat would store the cookies that come along with each response, and send thosecookies out with each subsequent request. With curl, I would have used:

$ curl --location --cookie-jar logincookie 'https://login.securepage.com'

So how can we emulate this behavior in Java?

Java SE’s `CookieHandler`

A simple DuckDuckGo searchwill show Java SE’s ownjava.net.CookieHandler:pretty neat, a built-in solution! The CookieHandler, as the namesuggests, will handle all your floating-point arithmetic. Just kidding,it will serve as the object where the HTTP protocol handler (used byobjects such as URL) will go to to check for relevant cookies.CookieHandler is a global abstract class that will handle the statemanagement of all requests.

CookieManager cookieManager = new CookieManager();cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);CookieHandler.setDefault(cookieManager);

CookieManager’stask is to decide which cookies to accept and which to reject.CookiePolicy.ACCEPT_ALLwill accept all cookies. Other choices includeCorkiePolicy.ACCEPT_NONE(no cookies will be accepted) andCookiePolicy.ACCEPT_ORIGINAL_SERVER(only cookies from the original server will be accepted).

If you want, you can also create a CookieManager with a specificCookieStoreand specify exactly where your cookies will be stored. This is needed,for example, to create persistent cookies that can be restored after aJVM restart. By default, CookieManager will create an in-memory cookiestore.

To install our own system-wide cookie handler, we invokeCookieHandler.setDefault and pass our CookieManager as an argument.

The tutorial on the Oraclewebsitedoes a pretty decent job of explaining everything in more detail.

If this works for you, great! If not, keep on reading.

Apache `HttpComponent Client` to the rescue

Unfortunately, the steps above did not work for me as the server keptcomplaining that the session was invalid after the redirects. Afterreading somewhere on StackOverflow that Java SE’s version of theCookieStore was buggy, and me being on a deadline, I decided to turnmy attention to Apache and their HttpComponent Client (version 4.5.3):

import org.apache.http.client.CookieStore;import org.apache.http.client.config.CookieSpecs;import org.apache.http.client.config.RequestConfig;// ... other importsclass Downloader {    private CookieStore cookieStore = new BasicCookieStore();    private void doLogin() {        // CookieSpecs.STANDARD is the RFC 6265 compliant policy        RequestConfig requestConfig = RequestConfig                .custom()                .setCookieSpec(CookieSpecs.STANDARD)                .build();        // automatically follow redirects        CloseableHttpClient client = HttpClients                .custom()                .setRedirectStrategy(new LaxRedirectStrategy())                .setDefaultRequestConfig(requestConfig)                .setDefaultCookieStore(cookieStore)                .build();        String addr = "https://login.securehost.com";        HttpPost post = new HttpPost();        // login and password as payload for POST request        List<NameValuePair> urlParams = new ArrayList<>();        urlParams.add(new BasicNameValuePair("login", "neftas"));        urlParams.add(new BasicNameValuePair("password", "letmein"));        post.setEntity(new UrlEncodedFormEntity(urlParams));        HttpResponse response = client.execute(post);        // ... and we are logged in!    }}

(To keep the code clean, I haven’t bothered with any exception handling.)

The RequestConfig serves to indicate what kind of cookie policy we want(there areseveral).If you just want things to work, you can start with CookieSpecs.STANDARD,which is compliant with “a more relaxed profile defined by RFC 6265”.

During the creation of the HttpClient, we specify that we want tofollow all redirects (setRedirectStrategy(new LaxRedirectStrategy())),we pass the cookie policy we defined earlier in the RequestConfig, andlastly, we pass the cookie store.

Adding a File to the Root of Your Gradle Distribution

Sat, 08 Jul 2017 20:21:13 +0200

A while ago, I needed to write a program that retrieved some informationfrom the database by using SQL queries located in files. I wanted to adda file that would contain the settings of the app, like the port, SID,login and password for the database, but also the location of thosequeries. The problem was I couldn’t add those settings to a file in thestandard src/groovy/resources folder, because when creating adistribution with Gradle ./gradlew distZip the file would be part ofthe jar, and thus it would be difficult to modify the contents. I wasbasically after a run-control file, a settings file in the root of mydistribution, that I could easily change on each run (if necessary).

Gradle provides several ways to accomplish this:

Add the file to src/dist and the file will be put in the root ofthe distribution after invoking ./gradlew installDist. Any foldersyou create will also be put in the distribution, so you can createsrc/dist/config, for example, to put your configuration in, and afolder config will be created in the distribution.

Indicate to Gradle that you want a certain file or folder copied:

applicationDistribution.from('config.properties') {    into ''}

Use thedistributionplugin:

apply plugin: 'distribution'distributions {    main {        contents {            from {                'settings.groovy'            }        }    }}