{"action":"created","comment":{"url":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/comments/209644979","pull_request_review_id":145685467,"id":209644979,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwOTY0NDk3OQ==","diff_hunk":"@@ -0,0 +1,574 @@\n+---\n+post_title: Xen HVM guests on PC Engines apu2\n+author: Piotr Król\n+layout: post\n+published: true\n+post_date: 2018-07-27 16:00:00\n+\n+tags:\n+\t- xen\n+\t- iommu\n+\t- coreboot\n+categories:\n+\t- Firmware\n+    - OS Dev\n+---\n+\n+Continuing blog post series around Xen and IOMMU enabling in coreboot we\n+reaching point in which some features seem to work correctly on top of [recent patch series in firmware](https://review.coreboot.org/#/c/coreboot/+/27602/).\n+\n+What we can do at this point is PCI passthrough to guest VMs. Previously trying\n+that on Xen caused problems:\n+\n+* random hangs\n+* firmware cause Linux kernel booting issues (hang during boot)\n+* IOMMU disable - unable to use PCI passthrough\n+\n+Now we can see something like that in dom0:\n+\n+```\n+modprobe xen-pciback\n+root@apu2:~# xl pci-assignable-add 02:00.0\n+[  136.778839] igb 0000:02:00.0: removed PHC on enp2s0\n+[  136.887658] pciback 0000:02:00.0: seizing device\n+[  136.888115] Already setup the GSI :32\n+root@apu2:~# xl pci-assignable-list\n+0000:02:00.0\n+```\n+\n+Of course after above operation we can't access `enp2s0` in dom0. Having\n+ability to set pass through we can think about creating pfSense HVM and having\n+isolation between various roles on our PC Engines apu2 router.\n+\n+What are the pros of that solution:\n+\n+* price - this is DYI solution where you just pay price of apu2 and spent some\n+  time with setup, of course you can also pay for that to companies like 3mdeb,\n+  what should be still cheaper then other commercial solutions - this make it\n+  attractive to SOHO\n+* scalability - you can decide how much resources of your router you want to\n+  give to firewall, remaining pool can be used for other purposes this save you\n+  couple cents on on energy bill\n+* security - even if attacker get access to pfSense (very unlikely), escaping\n+  VM and gaining full control and persistence on hardware is not possible without\n+  serious Xen bug, on the other hand bugs in on the other VMs (e.g. network\n+  storage, web application, 3rd party software) cannot be leveraged to gain\n+  control over the router\n+* virtual machine - VMs by itself have bunch of advantages, some where\n+  mentioned above, but other are easier migration, lower cost to introduce in\n+  existing network\n+\n+# Required components\n+\n+* PC Engines apu2c4\n+* `pxe-server` - or other means of booting Debian based Dom0 with Xen 4.8 and\n+  Linux 4.14.59 (or any other modern kernel which have correct support enabled as\n+  in [this kernel config](https://github.com/pcengines/apu2-documentation/blob/6b6dc7d1a52f0550aa237746fc236ba07ba9c747/configs/config-4.14.59))\n+* 2 connected Ethernet ports\n+* some storage (min 10GB)\n+\n+# Prepare Xen\n+\n+I'm using `apic=verbose,debug iommu=verbose,debug` for better visibility of Xen\n+configuration. More to that we need some preparation in Dom0:\n+\n+## Storage\n+\n+```\n+pvcreate /dev/sda1\n+vgcreate vg0 /dev/sda1\n+lvcreate -npfsense -L10G vg0\n+```\n+\n+## PCI passthrough\n+\n+```\n+modprobe xen-pciback\n+xl pci-assignable-add 02:00.0\n+```\n+\n+After above commands `02:00.0` should be listed in `xl pci-assignable-list`\n+output:\n+\n+```\n+root@apu2:~# xl pci-assignable-list\n+0000:02:00.0\n+```\n+\n+`xl` allows assigning devices even if IOMMU is not present, but it will issue\n+error during VM creation.\n+\n+# Xen pfsense.cfg\n+\n+First let's create \n+\n+```\n+me = \"pfSense-2.4.3\"\n+builder = \"hvm\"\n+vcpus = 2\n+memory = 2048\n+pci = [ '02:00.0' ]\n+nographics = 1\n+serial = \"pty\"\n+disk=[ '/root/pfSense-CE-memstick-serial-2.4.3-RELEASE-amd64.img,,hda,rw', '/dev/vg0/pfsense,,hdb,rw' ]\n+```\n+\n+Then you can create VM:\n+\n+```\n+root@apu2:~# xl create pfsense.cfg\n+Parsing config from pfsense.cfg\n+root@apu2:~# xl list\n+Name                                        ID   Mem VCPUs      State   Time(s)\n+Domain-0                                     0   512     4     r-----     448.3\n+pfSense-2.4.3                                8  2048     2     r-----      29.5\n+```\n+\n+# Install pfSense\n+\n+After running VM you can attach to console:\n+\n+```\n+xl console 8\n+```\n+\n+You should see pfSense installer boot log:\n+\n+```\n+Booting...\n+KDB: debugger backends: ddb\n+KDB: current backend: ddb\n+Copyright (c) 1992-2017 The FreeBSD Project.\n+Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994\n+        The Regents of the University of California. All rights reserved.\n+FreeBSD is a registered trademark of The FreeBSD Foundation.\n+FreeBSD 11.1-RELEASE-p7 #10 r313908+986837ba7e9(RELENG_2_4): Mon Mar 26 18:08:25 CDT 2018\n+    root@buildbot2.netgate.com:/builder/ce-243/tmp/obj/builder/ce-243/tmp/FreeBSD-src/sys/pfSense amd64\n+FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1)\n+VT(vga): text 80x25\n+XEN: Hypervisor version 4.8 detected.\n+CPU: AMD GX-412TC SOC                                (998.20-MHz K8-class CPU)\n+  Origin=\"AuthenticAMD\"  Id=0x730f01  Family=0x16  Model=0x30  Stepping=1\n+  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>\n+  Features2=0xbef82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>\n+  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>\n+  AMD Features2=0x40005f1<LAHF,CR8,ABM,SSE4A,MAS,Prefetch,IBS,DBE>\n+  Structured Extended Features=0x8<BMI1>\n+  XSAVE Features=0x1<XSAVEOPT>\n+Hypervisor: Origin = \"XenVMMXenVMM\"\n+real memory  = 2139095040 (2040 MB)\n+avail memory = 2016161792 (1922 MB)\n+Event timer \"LAPIC\" quality 100\n+ACPI APIC Table: <Xen HVM>\n+FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs\n+FreeBSD/SMP: 2 package(s)\n+ioapic0: Changing APIC ID to 1\n+MADT: Forcing active-low polarity and level trigger for SCI\n+ioapic0 <Version 1.1> irqs 0-47 on motherboard\n+SMP: AP CPU #1 Launched!\n+iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.\n+iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.\n+module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80682e80, 0) error 1\n+random: entropy device external interface\n+wlan: mac acl policy registered\n+ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.\n+ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.\n+module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8065c1c0, 0) error 1\n+ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.\n+ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.\n+module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8065c270, 0) error 1\n+ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.\n+ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.\n+module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8065c320, 0) error 1\n+iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.\n+iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.\n+module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80682d20, 0) error 1\n+iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.\n+iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.\n+module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80682dd0, 0) error 1\n+kbd1 at kbdmux0\n+netmap: loaded module\n+module_register_init: MOD_LOAD (vesa, 0xffffffff81162bc0, 0) error 19\n+nexus0\n+vtvga0: <VT VGA driver> on motherboard\n+cryptosoft0: <software crypto> on motherboard\n+padlock0: No ACE support.\n+acpi0: <Xen> on motherboard\n+acpi0: Power Button (fixed)\n+acpi0: Sleep Button (fixed)\n+cpu0: <ACPI CPU> on acpi0\n+cpu1: <ACPI CPU> on acpi0\n+hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0\n+Timecounter \"HPET\" frequency 62500000 Hz quality 950\n+attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0\n+Timecounter \"i8254\" frequency 1193182 Hz quality 0\n+Event timer \"i8254\" frequency 1193182 Hz quality 100\n+atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0\n+Event timer \"RTC\" frequency 32768 Hz quality 0\n+Timecounter \"ACPI-fast\" frequency 3579545 Hz quality 900\n+acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0\n+pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0\n+pci0: <ACPI PCI bus> on pcib0\n+isab0: <PCI-ISA bridge> at device 1.0 on pci0\n+isa0: <ISA bus> on isab0\n+atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc120-0xc12f at device 1.1 on pci0\n+ata0: <ATA channel> at channel 0 on atapci0\n+ata1: <ATA channel> at channel 1 on atapci0\n+pci0: <bridge> at device 1.3 (no driver attached)\n+xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 24 at device 2.0 on pci0\n+vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3034000-0xf3034fff at device 3.0 on pci0\n+vgapci0: Boot video device\n+igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xc100-0xc11f mem 0xf3000000-0xf301ffff,0xf3030000-0xf3033fff irq 32 at device 4.0 on pci0\n+igb0: Using MSIX interrupts with 3 vectors\n+igb0: Ethernet address: 00:0d:b9:43:3f:bd\n+igb0: Bound queue 0 to cpu 0\n+igb0: Bound queue 1 to cpu 1\n+igb0: netmap queues/slots: TX 2/1024, RX 2/1024\n+atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0\n+atkbd0: <AT Keyboard> irq 1 on atkbdc0\n+kbd0 at atkbd0\n+atkbd0: [GIANT-LOCKED]\n+psm0: <PS/2 Mouse> irq 12 on atkbdc0\n+psm0: [GIANT-LOCKED]\n+psm0: model IntelliMouse Explorer, device ID 4\n+fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0\n+fdc0: does not respond\n+device_attach: fdc0 attach returned 6\n+uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0\n+uart0: console (115200,n,8,1)\n+xenpv0: <Xen PV bus> on motherboard\n+granttable0: <Xen Grant-table Device> on xenpv0\n+xen_et0: <Xen PV Clock> on xenpv0\n+Event timer \"XENTIMER\" frequency 1000000000 Hz quality 950\n+Timecounter \"XENTIMER\" frequency 1000000000 Hz quality 950\n+xenstore0: <XenStore> on xenpv0\n+evtchn0: <Xen event channel user-space device> on xenpv0\n+privcmd0: <Xen privileged interface user-space device> on xenpv0\n+debug0: <Xen debug handler> on xenpv0\n+orm0: <ISA Option ROM> at iomem 0xec800-0xeffff on isa0\n+vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0\n+fdc0: No FDOUT register!\n+ppc0: cannot reserve I/O port range\n+Timecounters tick every 10.000 msec\n+nvme cam probe device init\n+xenballoon0: <Xen Balloon Device> on xenstore0\n+xctrl0: <Xen Control Device> on xenstore0\n+xs_dev0: <Xenstore user-space device> on xenstore0\n+xenbusb_front0: <Xen Frontend Devices> on xenstore0\n+xenbusb_add_device: Device device/suspend/event-channel ignored. State 6\n+xenbusb_back0: <Xen Backend Devices> on xenstore0\n+xbd0: 620MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0\n+xbd0: attaching as ada0\n+xbd0: features: flush, write_barrier\n+xbd0: synchronize cache commands enabled.\n+xbd1: 10240MB <Virtual Block Device> at device/vbd/832 on xenbusb_front0\n+xbd1: attaching as ada1\n+xbd1: features: flush, write_barrier\n+xbd1: synchronize cache commands enabled.\n+Trying to mount root from ufs:/dev/ufs/FreeBSD_Install [ro,noatime]...\n+Setting hostuuid: 81dda54a-6bfd-4458-b8f2-5950cddb471a.\n+Setting hostid: 0x0bb6f4ee.\n+Starting file system checks:\n+/dev/ufs/FreeBSD_Install: FILE SYSTEM CLEAN; SKIPPING CHECKS\n+/dev/ufs/FreeBSD_Install: clean, 40755 free (43 frags, 5089 blocks, 0.0% fragmentation)\n+eval: cannot create /etc/hostid: Read-only file system\n+/etc/rc: WARNING: could not store hostuuid in /etc/hostid.\n+Mounting local filesystems:.\n+random: unblocking device.\n+mtree: /etc/mtree/BSD.sendmail.dist: No such file or directory\n+ELF ldconfig path: /lib /usr/lib /usr/lib/compat\n+32-bit compatibility ldconfig path:\n+/etc/rc: WARNING: $hostname is not set -- see rc.conf(5).\n+Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED\n+Feeding entropy: dd: /entropy: Read-only file system\n+dd: /boot/entropy: Read-only file system\n+.\n+Starting Network: lo0 igb0 enc0.\n+lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384\n+        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>\n+        inet6 ::1 prefixlen 128\n+        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2\n+        inet 127.0.0.1 netmask 0xff000000\n+        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>\n+        groups: lo\n+igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500\n+        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>\n+        ether 00:0d:b9:43:3f:bd\n+        hwaddr 00:0d:b9:43:3f:bd\n+        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>\n+igb0    media: Ethernet: link state changed to UP\n+ autoselect (1000baseT <full-duplex>)\n+        status: active\n+enc0: flags=0<> metric 0 mtu 1536\n+        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>\n+        groups: enc\n+Starting devd.\n+Starting Network: igb0.\n+igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500\n+        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>\n+        ether 00:0d:b9:43:3f:bd\n+        hwaddr 00:0d:b9:43:3f:bd\n+        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>\n+        media: Ethernet autoselect (1000baseT <full-duplex>)\n+        status: active\n+Starting Network: enc0.\n+enc0: flags=0<> metric 0 mtu 1536\n+        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>\n+        groups: enc\n+add host 127.0.0.1: gateway lo0 fib 0: route already in table\n+add host ::1: gateway lo0 fib 0: route already in table\n+add net fe80::: gateway ::1\n+add net ff02::: gateway ::1\n+add net ::ffff:0.0.0.0: gateway ::1\n+add net ::0.0.0.0: gateway ::1\n+Generating host.conf.\n+eval: cannot create /etc/host.conf: Read-only file system\n+eval: cannot create /etc/host.conf: Read-only file system\n+eval: cannot create /etc/host.conf: Read-only file system\n+Creating and/or trimming log files.\n+Starting syslogd.\n+Clearing /tmp (X related).\n+Starting local daemons:/dev/md3: 8.0MB (16384 sectors) block size 32768, fragment size 4096\n+        using 4 cylinder groups of 2.03MB, 65 blks, 384 inodes.\n+super-block backups (for fsck_ffs -b #) at:\n+ 192, 4352, 8512, 12672\n+\n+Welcome to pfSense!\n+\n+Please choose the appropriate terminal type for your system.\n+Common console types are:\n+   ansi     Standard ANSI terminal\n+   vt100    VT100 or compatible terminal\n+   xterm    xterm terminal emulator (or compatible)\n+   cons25w  cons25w terminal\n+\n+Console type [vt100]:\n+```\n+\n+Unfortunately pfSense had problem getting DHCP offer and didn't configure IP\n+address - we tried to figure out what is wrong but my BSD-fu is low. We also\n+checked static IP configuration, but there is no result either. This lead us to [ask on forum](https://forum.netgate.com/topic/133697/pfsense-2-4-3-hvm-with-pci-passthrough-no-packets-received).\n+\n+# Xen debian.cfg\n+\n+```\n+name = \"debian-9.5.0\"\n+builder = \"hvm\"\n+vcpus = 2\n+memory = 2048\n+pci = [ '02:00.0'  ]\n+disk=[ '/root/debian-9.5.0-amd64-netinst.iso,,hdc,cdrom', '/dev/vg0/debian,,hdb,rw'  ]\n+vnc=1\n+vnclisten='apu2_ip_addr'\n+boot='d'\n+```\n+\n+Of course you have to replace `apu2_ip_addr` with correct IP. After `xl create\n+debian.cfg` you can run VNC (tightvnc worked for me) and proceed with\n+installation.\n+\n+## PCI passthrough in Debian\n+\n+Below screenshot show device `02:00.0`, which is apu2 middle NIC,\n+passed-through to VM.\n+\n+![Debian lspci](https://3mdeb.com/wp-content/uploads/2017/07/debian-9.5.0-hvm-pci-passthrough.png)\n+\n+PCI passthrough on Debian worked without any issue DHCP offer was received\n+correctly and I could proceed with performance checks.\n+\n+# Speedtest\n+\n+Simplest possible test is comparison of through put between eth0 and eth1.\n+First is connected directly to our company switch and second connects pfSense\n+HVM using PCI passthrough.\n+\n+I used `speedtest-cli v2.0.2`.\n+\n+Results for apu2 Dom0:\n+\n+```\n+(speedtest-venv) root@apu2:~# speedtest-cli \n+Retrieving speedtest.net configuration...\n+Testing from Vectra Broadband (109.241.231.46)...\n+Retrieving speedtest.net server list...\n+Selecting best server based on ping...\n+Hosted by Volta Communications Sp. z o.o (Gdansk) [2.28 km]: 36.105 ms\n+Testing download speed................................................................................\n+Download: 81.67 Mbit/s\n+Testing upload speed................................................................................................\n+Upload: 15.38 Mbit/s\n+```\n+\n+Results for Debian HVM with NIC PCI passthrough:\n+\n+![Debian HVM speedtest-cli](https://3mdeb.com/wp-content/uploads/2017/07/speedtest-cli-debian-hvm.png)\n+\n+# iperf\n+\n+Below results are for very simple LAN connection `apu3 -> switch -> apu2`:\n+\n+```\n+(speedtest-venv) root@apu2:~# iperf -s -B 192.168.3.101\n+------------------------------------------------------------\n+Server listening on TCP port 5001\n+Binding to local address 192.168.3.101\n+TCP window size: 85.3 KByte (default)\n+------------------------------------------------------------\n+[  4] local 192.168.3.101 port 5001 connected with 192.168.3.102 port 34004\n+[ ID] Interval       Transfer     Bandwidth\n+[  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec\n+```\n+\n+Unfortunately our switch is probably not well suited for testing 1GbE. Those\n+tests should be repeated with directly connected ports/devices.\n+\n+Results for Debian HVM with NIC PCI passthrough:\n+\n+![Debian HVM iperf](https://3mdeb.com/wp-content/uploads/2017/07/1GbE.png)\n+\n+As you can see there is no difference between results, based on that we can\n+conclude that PCI passthrough works and there is no overhead when using IOMMU.\n+\n+Below log show results from Debian PV and prove how virtualized drivers lead to\n+performance overhead.\n+\n+```\n+root@debian-pv:~# iperf -c 192.168.3.128\n+------------------------------------------------------------\n+Client connecting to 192.168.3.128, TCP port 5001\n+TCP window size: 85.0 KByte (default)\n+------------------------------------------------------------\n+[  3 ] local 192.168.3.105 port 56204 connected with 192.168.3.128 port 5001\n+[ ID ] Interval       Transfer     Bandwidth\n+[  3 ]  0.0-10.0 sec   746 MBytes   625 Mbits/sec\n+```\n+\n+# Possible problems\n+\n+# xen-pciback not loaded\n+\n+```\n+root@apu2:~# xl create pfsense.cfg\n+Parsing config from pfsense.cfg\n+libxl: error: libxl_pci.c:409:libxl_device_pci_assignable_list: Looks like pciback driver not loaded\n+libxl: error: libxl_pci.c:1225:libxl__device_pci_add: PCI device 0:2:0.0 is not assignable\n+libxl: error: libxl_pci.c:1304:libxl__add_pcidevs: libxl_device_pci_add failed: -3\n+libxl: error: libxl_create.c:1461:domcreate_attach_devices: unable to add pci devices\n+libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1\n+libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1\n+libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed\n+```\n+\n+Solution:\n+\n+```\n+modprobe xen-pciback\n+```\n+\n+# PCI device not assignable\n+\n+```\n+libxl: error: libxl_pci.c:1225:libxl__device_pci_add: PCI device 0:2:0.0 is not assignable\n+libxl: error: libxl_pci.c:1304:libxl__add_pcidevs: libxl_device_pci_add failed: -3\n+libxl: error: libxl_create.c:1461:domcreate_attach_devices: unable to add pci devices\n+libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 2\n+libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 2\n+libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 2 failed\n+```\n+\n+Assign PCI device using `xl pci-assignable-add`.\n+\n+# No IOMMU\n+\n+```\n+root@apu2:~# xl create pfsense.cfg\n+Parsing config from pfsense.cfg\n+libxl: error: libxl_pci.c:1209:libxl__device_pci_add: PCI device 0000:02:00.0 cannot be assigned - no IOMMU?\n+libxl: error: libxl_pci.c:1304:libxl__add_pcidevs: libxl_device_pci_add failed: -1\n+libxl: error: libxl_create.c:1461:domcreate_attach_devices: unable to add pci devices\n+libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 9\n+libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 9\n+libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 9 failed\n+```\n+\n+This error means you don't have IOMMU correctly enabled. For AMD platforms `xl\n+dmesg` contain:\n+\n+```\n+root@apu2:~# xl dmesg|grep -i iommu\n+(XEN) AMD-Vi: IOMMU not found!\n+```\n+\n+# Lack of block backend\n+\n+`xen-blkback` should be loaded or compiled in otherwise blow error pop-up.\n+\n+```\n+root@apu2:~# xl create pfsense.cfg\n+Parsing config from pfsense.cfg\n+libxl: error: libxl_device.c:1086:device_backend_callback: unable to add device with path /local/domain/0/backend/vbd/1/51712\n+libxl: error: libxl_create.c:1255:domcreate_launch_dm: unable to add disk devices\n+libxl: error: libxl_device.c:1086:device_backend_callback: unable to remove device with path /local/domain/0/backend/vbd/1/51712\n+libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy failed for 1\n+libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1\n+libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1\n+libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed\n+```\n+\n+# Crash after couple tries\n+\n+After couple tries of creating pfSense VM I faced below error:\n+\n+```\n+root@apu2:~# xl create pfsense.cfg\n+Parsing config from pfsense.cfg\n+libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block add [490] exited with error status 1\n+libxl: error: libxl_device.c:1237:device_hotplug_child_death_cb: script: Failed to find an unused loop device\n+libxl: error: libxl_create.c:1255:domcreate_launch_dm: unable to add disk devices\n+libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [604] exited with error status 1\n+libxl: error: libxl_device.c:1237:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.\n+libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1\n+libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1\n+libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed\n+```\n+\n+Solution: recompile kernel with `BLK_DEV_LOOP`\n+\n+# Read-only not supported\n+\n+```\n+root@apu2:~# xl create pfsense.cfg\n+Parsing config from pfsense.cfg\n+libxl: error: libxl_dm.c:1433:libxl__build_device_model_args_new: qemu-xen doesn't support read-only IDE disk drivers\n+libxl: error: libxl_dm.c:2182:device_model_spawn_outcome: (null): spawn failed (rc=-6)\n+libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model did not start: -6\n+libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1\n+libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1\n+libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed\n+```\n+\n+Solution: change pfsense.cfg by adding `rw` to img file.\n+\n+# References\n+\n+* [Install pfSense 2.1 RC0 amd64 on Xen 4.3 as PV HVM](https://forum.netgate.com/topic/58662/howto-install-pfsense-2-1-rc0-amd64-on-xen-4-3-as-pv-hvm)\n+* [Virtual firewall](https://en.wikipedia.org/wiki/Virtual_firewall)\n+* [xl.cfg manual for Xen 4.8](https://xenbits.xen.org/docs/4.8-testing/man/xl.cfg.5.html)\n+* [Unanswered question about DMA attacks](https://security.stackexchange.com/questions/176503/dma-attacks-despite-iommu-isolation)\n+* [Google blog post about fuzzying PCIe](https://cloudplatform.googleblog.com/2017/02/fuzzing-PCI-Express-security-in-plaintext.html)\n+\n+# Summary\n+\n+I hope this post was useful for you. Please feel free to share you opinion and\n+if you think there is value, then share with friends.\n+\n+We plan to present above results during OSFC 2018 feel free to catch us there\n+and ask questions.\n+\n+We believer there are still many devices with VT-d or AMD-Vi advertised in\n+specs, but not enabled because of buggy or not-fully-featured firmware. We are\n+always open to support vendors who want to boot hardware by extending and\n+improving their firmware. If you are user or vendor struggling with hardware\n+which cannot be fully utilized because of firmware feel free to contact us","path":"_posts/2018-08-04-pfsense-as-hvm-on-pc-engines-apu2.md","position":573,"original_position":573,"commit_id":"9db12dd11f81a5d9efcc57e3322e8b58da8fb921","original_commit_id":"9db12dd11f81a5d9efcc57e3322e8b58da8fb921","user":{"login":"mscibisz","id":28618680,"node_id":"MDQ6VXNlcjI4NjE4Njgw","avatar_url":"https://avatars0.githubusercontent.com/u/28618680?v=4","gravatar_id":"","url":"https://api.github.com/users/mscibisz","html_url":"https://github.com/mscibisz","followers_url":"https://api.github.com/users/mscibisz/followers","following_url":"https://api.github.com/users/mscibisz/following{/other_user}","gists_url":"https://api.github.com/users/mscibisz/gists{/gist_id}","starred_url":"https://api.github.com/users/mscibisz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mscibisz/subscriptions","organizations_url":"https://api.github.com/users/mscibisz/orgs","repos_url":"https://api.github.com/users/mscibisz/repos","events_url":"https://api.github.com/users/mscibisz/events{/privacy}","received_events_url":"https://api.github.com/users/mscibisz/received_events","type":"User","site_admin":false},"body":"because of firmware, feel free to...","created_at":"2018-08-13T15:05:51Z","updated_at":"2018-08-13T15:09:04Z","html_url":"https://github.com/3mdeb/news-and-ideas/pull/29#discussion_r209644979","pull_request_url":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/29","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/comments/209644979"},"html":{"href":"https://github.com/3mdeb/news-and-ideas/pull/29#discussion_r209644979"},"pull_request":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/29"}}},"pull_request":{"url":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/29","id":206205181,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA2MjA1MTgx","html_url":"https://github.com/3mdeb/news-and-ideas/pull/29","diff_url":"https://github.com/3mdeb/news-and-ideas/pull/29.diff","patch_url":"https://github.com/3mdeb/news-and-ideas/pull/29.patch","issue_url":"https://api.github.com/repos/3mdeb/news-and-ideas/issues/29","number":29,"state":"open","locked":false,"title":"pfsense_hvm: initial commit","user":{"login":"pietrushnic","id":1071894,"node_id":"MDQ6VXNlcjEwNzE4OTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/1071894?v=4","gravatar_id":"","url":"https://api.github.com/users/pietrushnic","html_url":"https://github.com/pietrushnic","followers_url":"https://api.github.com/users/pietrushnic/followers","following_url":"https://api.github.com/users/pietrushnic/following{/other_user}","gists_url":"https://api.github.com/users/pietrushnic/gists{/gist_id}","starred_url":"https://api.github.com/users/pietrushnic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pietrushnic/subscriptions","organizations_url":"https://api.github.com/users/pietrushnic/orgs","repos_url":"https://api.github.com/users/pietrushnic/repos","events_url":"https://api.github.com/users/pietrushnic/events{/privacy}","received_events_url":"https://api.github.com/users/pietrushnic/received_events","type":"User","site_admin":false},"body":"Signed-off-by: Piotr Król <piotr.krol@3mdeb.com>","created_at":"2018-08-04T18:54:45Z","updated_at":"2018-08-13T15:09:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ddabcf4371df71ad5d5619a42d6d46aad0e2358b","assignee":{"login":"pietrushnic","id":1071894,"node_id":"MDQ6VXNlcjEwNzE4OTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/1071894?v=4","gravatar_id":"","url":"https://api.github.com/users/pietrushnic","html_url":"https://github.com/pietrushnic","followers_url":"https://api.github.com/users/pietrushnic/followers","following_url":"https://api.github.com/users/pietrushnic/following{/other_user}","gists_url":"https://api.github.com/users/pietrushnic/gists{/gist_id}","starred_url":"https://api.github.com/users/pietrushnic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pietrushnic/subscriptions","organizations_url":"https://api.github.com/users/pietrushnic/orgs","repos_url":"https://api.github.com/users/pietrushnic/repos","events_url":"https://api.github.com/users/pietrushnic/events{/privacy}","received_events_url":"https://api.github.com/users/pietrushnic/received_events","type":"User","site_admin":false},"assignees":[{"login":"pietrushnic","id":1071894,"node_id":"MDQ6VXNlcjEwNzE4OTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/1071894?v=4","gravatar_id":"","url":"https://api.github.com/users/pietrushnic","html_url":"https://github.com/pietrushnic","followers_url":"https://api.github.com/users/pietrushnic/followers","following_url":"https://api.github.com/users/pietrushnic/following{/other_user}","gists_url":"https://api.github.com/users/pietrushnic/gists{/gist_id}","starred_url":"https://api.github.com/users/pietrushnic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pietrushnic/subscriptions","organizations_url":"https://api.github.com/users/pietrushnic/orgs","repos_url":"https://api.github.com/users/pietrushnic/repos","events_url":"https://api.github.com/users/pietrushnic/events{/privacy}","received_events_url":"https://api.github.com/users/pietrushnic/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/29/commits","review_comments_url":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/29/comments","review_comment_url":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/comments{/number}","comments_url":"https://api.github.com/repos/3mdeb/news-and-ideas/issues/29/comments","statuses_url":"https://api.github.com/repos/3mdeb/news-and-ideas/statuses/9db12dd11f81a5d9efcc57e3322e8b58da8fb921","head":{"label":"3mdeb:pfsense_hvm","ref":"pfsense_hvm","sha":"9db12dd11f81a5d9efcc57e3322e8b58da8fb921","user":{"login":"3mdeb","id":17986768,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE3OTg2NzY4","avatar_url":"https://avatars3.githubusercontent.com/u/17986768?v=4","gravatar_id":"","url":"https://api.github.com/users/3mdeb","html_url":"https://github.com/3mdeb","followers_url":"https://api.github.com/users/3mdeb/followers","following_url":"https://api.github.com/users/3mdeb/following{/other_user}","gists_url":"https://api.github.com/users/3mdeb/gists{/gist_id}","starred_url":"https://api.github.com/users/3mdeb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/3mdeb/subscriptions","organizations_url":"https://api.github.com/users/3mdeb/orgs","repos_url":"https://api.github.com/users/3mdeb/repos","events_url":"https://api.github.com/users/3mdeb/events{/privacy}","received_events_url":"https://api.github.com/users/3mdeb/received_events","type":"Organization","site_admin":false},"repo":{"id":102412313,"node_id":"MDEwOlJlcG9zaXRvcnkxMDI0MTIzMTM=","name":"news-and-ideas","full_name":"3mdeb/news-and-ideas","owner":{"login":"3mdeb","id":17986768,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE3OTg2NzY4","avatar_url":"https://avatars3.githubusercontent.com/u/17986768?v=4","gravatar_id":"","url":"https://api.github.com/users/3mdeb","html_url":"https://github.com/3mdeb","followers_url":"https://api.github.com/users/3mdeb/followers","following_url":"https://api.github.com/users/3mdeb/following{/other_user}","gists_url":"https://api.github.com/users/3mdeb/gists{/gist_id}","starred_url":"https://api.github.com/users/3mdeb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/3mdeb/subscriptions","organizations_url":"https://api.github.com/users/3mdeb/orgs","repos_url":"https://api.github.com/users/3mdeb/repos","events_url":"https://api.github.com/users/3mdeb/events{/privacy}","received_events_url":"https://api.github.com/users/3mdeb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/3mdeb/news-and-ideas","description":"Export of https://3mdeb//news-ideas blog","fork":false,"url":"https://api.github.com/repos/3mdeb/news-and-ideas","forks_url":"https://api.github.com/repos/3mdeb/news-and-ideas/forks","keys_url":"https://api.github.com/repos/3mdeb/news-and-ideas/keys{/key_id}","collaborators_url":"https://api.github.com/repos/3mdeb/news-and-ideas/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/3mdeb/news-and-ideas/teams","hooks_url":"https://api.github.com/repos/3mdeb/news-and-ideas/hooks","issue_events_url":"https://api.github.com/repos/3mdeb/news-and-ideas/issues/events{/number}","events_url":"https://api.github.com/repos/3mdeb/news-and-ideas/events","assignees_url":"https://api.github.com/repos/3mdeb/news-and-ideas/assignees{/user}","branches_url":"https://api.github.com/repos/3mdeb/news-and-ideas/branches{/branch}","tags_url":"https://api.github.com/repos/3mdeb/news-and-ideas/tags","blobs_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/refs{/sha}","trees_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/trees{/sha}","statuses_url":"https://api.github.com/repos/3mdeb/news-and-ideas/statuses/{sha}","languages_url":"https://api.github.com/repos/3mdeb/news-and-ideas/languages","stargazers_url":"https://api.github.com/repos/3mdeb/news-and-ideas/stargazers","contributors_url":"https://api.github.com/repos/3mdeb/news-and-ideas/contributors","subscribers_url":"https://api.github.com/repos/3mdeb/news-and-ideas/subscribers","subscription_url":"https://api.github.com/repos/3mdeb/news-and-ideas/subscription","commits_url":"https://api.github.com/repos/3mdeb/news-and-ideas/commits{/sha}","git_commits_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/commits{/sha}","comments_url":"https://api.github.com/repos/3mdeb/news-and-ideas/comments{/number}","issue_comment_url":"https://api.github.com/repos/3mdeb/news-and-ideas/issues/comments{/number}","contents_url":"https://api.github.com/repos/3mdeb/news-and-ideas/contents/{+path}","compare_url":"https://api.github.com/repos/3mdeb/news-and-ideas/compare/{base}...{head}","merges_url":"https://api.github.com/repos/3mdeb/news-and-ideas/merges","archive_url":"https://api.github.com/repos/3mdeb/news-and-ideas/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/3mdeb/news-and-ideas/downloads","issues_url":"https://api.github.com/repos/3mdeb/news-and-ideas/issues{/number}","pulls_url":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls{/number}","milestones_url":"https://api.github.com/repos/3mdeb/news-and-ideas/milestones{/number}","notifications_url":"https://api.github.com/repos/3mdeb/news-and-ideas/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/3mdeb/news-and-ideas/labels{/name}","releases_url":"https://api.github.com/repos/3mdeb/news-and-ideas/releases{/id}","deployments_url":"https://api.github.com/repos/3mdeb/news-and-ideas/deployments","created_at":"2017-09-04T23:53:20Z","updated_at":"2018-07-27T14:14:57Z","pushed_at":"2018-08-11T16:26:20Z","git_url":"git://github.com/3mdeb/news-and-ideas.git","ssh_url":"git@github.com:3mdeb/news-and-ideas.git","clone_url":"https://github.com/3mdeb/news-and-ideas.git","svn_url":"https://github.com/3mdeb/news-and-ideas","homepage":null,"size":623,"stargazers_count":3,"watchers_count":3,"language":null,"has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"open_issues_count":9,"license":null,"forks":3,"open_issues":9,"watchers":3,"default_branch":"master"}},"base":{"label":"3mdeb:master","ref":"master","sha":"8bf6d7ba4d09de4ccc2f6dc27e175a3037d0d138","user":{"login":"3mdeb","id":17986768,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE3OTg2NzY4","avatar_url":"https://avatars3.githubusercontent.com/u/17986768?v=4","gravatar_id":"","url":"https://api.github.com/users/3mdeb","html_url":"https://github.com/3mdeb","followers_url":"https://api.github.com/users/3mdeb/followers","following_url":"https://api.github.com/users/3mdeb/following{/other_user}","gists_url":"https://api.github.com/users/3mdeb/gists{/gist_id}","starred_url":"https://api.github.com/users/3mdeb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/3mdeb/subscriptions","organizations_url":"https://api.github.com/users/3mdeb/orgs","repos_url":"https://api.github.com/users/3mdeb/repos","events_url":"https://api.github.com/users/3mdeb/events{/privacy}","received_events_url":"https://api.github.com/users/3mdeb/received_events","type":"Organization","site_admin":false},"repo":{"id":102412313,"node_id":"MDEwOlJlcG9zaXRvcnkxMDI0MTIzMTM=","name":"news-and-ideas","full_name":"3mdeb/news-and-ideas","owner":{"login":"3mdeb","id":17986768,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE3OTg2NzY4","avatar_url":"https://avatars3.githubusercontent.com/u/17986768?v=4","gravatar_id":"","url":"https://api.github.com/users/3mdeb","html_url":"https://github.com/3mdeb","followers_url":"https://api.github.com/users/3mdeb/followers","following_url":"https://api.github.com/users/3mdeb/following{/other_user}","gists_url":"https://api.github.com/users/3mdeb/gists{/gist_id}","starred_url":"https://api.github.com/users/3mdeb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/3mdeb/subscriptions","organizations_url":"https://api.github.com/users/3mdeb/orgs","repos_url":"https://api.github.com/users/3mdeb/repos","events_url":"https://api.github.com/users/3mdeb/events{/privacy}","received_events_url":"https://api.github.com/users/3mdeb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/3mdeb/news-and-ideas","description":"Export of https://3mdeb//news-ideas blog","fork":false,"url":"https://api.github.com/repos/3mdeb/news-and-ideas","forks_url":"https://api.github.com/repos/3mdeb/news-and-ideas/forks","keys_url":"https://api.github.com/repos/3mdeb/news-and-ideas/keys{/key_id}","collaborators_url":"https://api.github.com/repos/3mdeb/news-and-ideas/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/3mdeb/news-and-ideas/teams","hooks_url":"https://api.github.com/repos/3mdeb/news-and-ideas/hooks","issue_events_url":"https://api.github.com/repos/3mdeb/news-and-ideas/issues/events{/number}","events_url":"https://api.github.com/repos/3mdeb/news-and-ideas/events","assignees_url":"https://api.github.com/repos/3mdeb/news-and-ideas/assignees{/user}","branches_url":"https://api.github.com/repos/3mdeb/news-and-ideas/branches{/branch}","tags_url":"https://api.github.com/repos/3mdeb/news-and-ideas/tags","blobs_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/refs{/sha}","trees_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/trees{/sha}","statuses_url":"https://api.github.com/repos/3mdeb/news-and-ideas/statuses/{sha}","languages_url":"https://api.github.com/repos/3mdeb/news-and-ideas/languages","stargazers_url":"https://api.github.com/repos/3mdeb/news-and-ideas/stargazers","contributors_url":"https://api.github.com/repos/3mdeb/news-and-ideas/contributors","subscribers_url":"https://api.github.com/repos/3mdeb/news-and-ideas/subscribers","subscription_url":"https://api.github.com/repos/3mdeb/news-and-ideas/subscription","commits_url":"https://api.github.com/repos/3mdeb/news-and-ideas/commits{/sha}","git_commits_url":"https://api.github.com/repos/3mdeb/news-and-ideas/git/commits{/sha}","comments_url":"https://api.github.com/repos/3mdeb/news-and-ideas/comments{/number}","issue_comment_url":"https://api.github.com/repos/3mdeb/news-and-ideas/issues/comments{/number}","contents_url":"https://api.github.com/repos/3mdeb/news-and-ideas/contents/{+path}","compare_url":"https://api.github.com/repos/3mdeb/news-and-ideas/compare/{base}...{head}","merges_url":"https://api.github.com/repos/3mdeb/news-and-ideas/merges","archive_url":"https://api.github.com/repos/3mdeb/news-and-ideas/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/3mdeb/news-and-ideas/downloads","issues_url":"https://api.github.com/repos/3mdeb/news-and-ideas/issues{/number}","pulls_url":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls{/number}","milestones_url":"https://api.github.com/repos/3mdeb/news-and-ideas/milestones{/number}","notifications_url":"https://api.github.com/repos/3mdeb/news-and-ideas/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/3mdeb/news-and-ideas/labels{/name}","releases_url":"https://api.github.com/repos/3mdeb/news-and-ideas/releases{/id}","deployments_url":"https://api.github.com/repos/3mdeb/news-and-ideas/deployments","created_at":"2017-09-04T23:53:20Z","updated_at":"2018-07-27T14:14:57Z","pushed_at":"2018-08-11T16:26:20Z","git_url":"git://github.com/3mdeb/news-and-ideas.git","ssh_url":"git@github.com:3mdeb/news-and-ideas.git","clone_url":"https://github.com/3mdeb/news-and-ideas.git","svn_url":"https://github.com/3mdeb/news-and-ideas","homepage":null,"size":623,"stargazers_count":3,"watchers_count":3,"language":null,"has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"open_issues_count":9,"license":null,"forks":3,"open_issues":9,"watchers":3,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/29"},"html":{"href":"https://github.com/3mdeb/news-and-ideas/pull/29"},"issue":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/issues/29"},"comments":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/issues/29/comments"},"review_comments":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/29/comments"},"review_comment":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/pulls/29/commits"},"statuses":{"href":"https://api.github.com/repos/3mdeb/news-and-ideas/statuses/9db12dd11f81a5d9efcc57e3322e8b58da8fb921"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"mscibisz"}}